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EXECUTIVE SUMMARY 


This report summarizes analyses performed on IDEA student ratings of instruction 
collected in traditional (i.e., on-campus) and online courses from 2002 to 2008. Only classes 
utilizing online survey delivery and identified exclusively as either traditional (N = 5,272) or 
online (N = 13,416) were included. Classes were removed until all institutions contributed no 
more than approximately 5% of all classes analyzed. Instructors in each course rated the 
importance of each of 12 learning objectives and responded to questions about the course, using 
the IDEA Faculty Information Form (FIF). Students rated progress on the same 12 objectives, 
characteristics about themselves and the course, and overall measures of course and instructor 
effectiveness, using either the IDEA Diagnostic Form or Short Form. Those responding to the 
Diagnostic Form also rated the instructor’s use of various teaching methods. 


Comparisons made between traditional and online courses revealed the following 
similarities. First, online instructor ratings of the importance of the 12 learning objectives 
paralleled those of on-campus instructors. No meaningful differences were found in instructors’ 
average ratings of importance and in the percent of instructors rating each objective as essential 
or important. Instructors, therefore, found the objectives relevant at similar levels in both on- 
campus and online courses. Second, students’ ratings of progress on relevant objectives and 
global measures of course and instructor excellence were very similar. Good teaching was good 
teaching, regardless of the course modality. Third, students consistently reported greater progress 
on objectives the instructor rated as important or essential regardless of the course format. 
Furthermore, the highest correlations between instructor ratings of importance and students’ 
ratings of progress were consistently observed in ratings of the same objectives. These findings 
support the validity of IDEA in both traditional and online environments. Fourth, student ratings 
of how frequently the instructor used each of 20 teaching methods did not vary meaningfully 
between the type of course. Instructors in online courses were perceived to use the various 
teaching methods as frequently as those in traditional courses. Fifth, the pattern of correlations 
between students’ ratings of progress on relevant objectives and the instructor’s use of teaching 
methods was also similar in traditional and online courses. Formative evaluations, based on these 
relationships, therefore have similar validity across modalities. Sixth, correlations between 
student/course characteristics and global measures of effectiveness were highly comparable. 
Adjustments to raw scores, based on these relationships, therefore have similar validity in both 
course formats. 


Some minor differences were observed, however. First, student response rate was higher 
in traditional than online courses. When using IDEA Online, administrators and instructors 
should consider employing best practices for online response rates (see 
http://www.theideacenter.org/OnlineResponseRates). Second, students in online courses reported 
more frequent instructor use of educational technology to promote student learning than did 
those in traditional courses. Moreover, instructor use of educational technology was more highly 
correlated with student progress on relevant objectives in online than in traditional courses. 
These differences make sense because, by their very nature, online courses rely heavily upon 
educational technology. However, the differences do not affect course evaluations because 
instructor use of educational technology does not affect summative or formative scores on the 
Diagnostic and Short Form reports. Third, students in online courses reported somewhat more 


reading and somewhat less motivation to take the course from the instructor. However, neither of 
these variables figures prominently in any raw score adjustments. Fourth, structuring the 
classroom environment may be somewhat more important in online courses, if the instructor 
wishes to help students achieve a broad liberal education. Finally, when helping students to find 
and use resources, establishing rapport may be somewhat more important in online courses. 


In general, the current findings indicate the IDEA Student Ratings System is useful for 
both online and traditional courses. The minor differences observed ultimately may guide 
instructors to improve student learning outcomes in online teaching environments. 


However, The IDEA Center recognizes that no single survey can anticipate the unique 
needs of every learning environment. The use of additional questions may be helpful in 
addressing areas not covered in the IDEA instrument that are important to a particular course or 
learning environment. Appendix B contains the handout, Using Additional Questions for Online 
Learning Environments, which can serve as a guide to maximizing the feedback obtained 
through the IDEA Student Ratings of Instruction System. 


An Analysis of IDEA Student Ratings in 
Online Versus Traditional Courses 
2002-2008 Data 


The purpose of this report is to summarize results from statistical analyses comparing 
online and traditional courses that completed the IDEA Student Ratings instrument from 2002 to 
2008. Only courses utilizing online survey delivery were included. Comparisons were made on 
students’ response rates and on instructor and student ratings. Since the IDEA system has 
historically been used to provide instructional feedback in the traditional classroom environment, 
the current investigation was undertaken to determine if the instrument is appropriate for use in 
an online learning environment. Several questions guided the analyses: 


1. Do student response rates to IDEA Online differ between traditional and online 
courses? 

a Do instructor ratings of the importance of the 12 IDEA learning objectives 
differ between types of courses? 

3. Are the inter-correlations among the instructor ratings of the 12 learning 
objectives similar across types of courses? 

4. Are there differences in students’ ratings of progress on the 12 learning 
objectives? 

5. Do students’ ratings of how frequently the instructor used various teaching 
methods differ between traditional and online courses? 

6. Are the correlations between instructors’ and students’ ratings of learning 
objectives similar in traditional and online environments? 

re Are the correlations between students’ ratings of progress on learning 


objectives and their ratings of the instructor’s teaching methods similar 
between types of courses? 

8. Are the correlations between students’ characteristics (e.g., work habits, 
motivation), overall measures of the course and the instructor, and perceived 
progress on relevant objectives similar across these course modalities? 

o: Are the correlations between students’ ratings of teaching methods and overall 
global ratings similar across types of courses? 


METHOD 


Sample 


The sample of classes was taken from 2002 to 2008 IDEA surveys administered through the 
IDEA Online survey delivery system. Not all classes that use IDEA Online are considered 
“online courses”; therefore, staff at The IDEA Center contacted users individually to ascertain 
whether their courses were taught on campus (traditional), via the Internet (online), or in some 
combination. This report included only classes identified exclusively as traditional or online. 
Prior to conducting the analyses, classes were removed until all institutions contributed no more 


than approximately 5% of all classes.' Although 73,514 classes administered online surveys, 
accurate course delivery information was only available for 18,688 classes. We were unable to 
designate 38,049 as either exclusively traditional or online, and numerous classes were 
eliminated to conform to the 5% institutional criterion. A total of 5,272 classes were identified as 
traditional (the course was conducted on campus), and 13,416 were determined to be online 
courses. Table | presents the frequency and percentage of classes coded as either traditional or 
online across the seven-year period. In this sample of classes, the proportion of traditional 
courses using IDEA Online increased across the years. Initially, online survey delivery was used 
almost exclusively for online courses. Over time, campuses have shifted to administering surveys 
online for traditional, on-campus courses as well. 


Table 1 
Frequency and Percentage of Classes Using IDEA Online 
Disaggregated by Year and Type of Course Instruction (Traditional vs. Online) 


Type of Course Instruction 
Traditional Online 

Year N % N % Total 
2002 15 6.5% 216 93.5% 231 
2003 30 9.2% 296 90.8% 326 
2004 109 23.4% 357 76.6% 466 
2005 355 32.1% 750 67.9% 1,105 
2006 754 28.1% 1,932 71.9% 2,686 
2007 1,032 22.8% 3,504 77.2% 4,536 
2008 2917. 31.9% 6,361 68.1% 9,338 
Total 5,272 28.2% 13,416 71.8% 18,688 


Table 2 presents the frequency and percentage of traditional and online classes, 
respectively, by the highest degrees awarded. There were 38 institutions represented in the 
traditional group and 67 in the online group. Table 2 also presents the frequency and percentage 
of highest degrees awarded in the overall IDEA database and among users of IDEA Online. In 
both traditional and online classes, the percentage of institutions offering the respective degrees 
was, in most cases, similar to that in the 2002 to 2008 overall IDEA database. There were two 
notable exceptions. The online courses in the current sample slightly underrepresented 
institutions offering the baccalaureate as the highest degree. Classes in the traditional-course 
group somewhat underrepresented institutions offering degrees beyond the master’s. However, 
as reported in Technical Report 12, IDEA students ratings do not differ by the highest degree 
awarded (Hoyt and Lee, 2002a). The current samples of classes are representative of the overall 
IDEA database and all ratings administered online. 


' Not all exclusion criteria from “IDEA Technical Report 12” were enacted. Removing classes with < 10 
respondents, classes using the short form, and novice classes would have dramatically reduced the number of 
available classes (Traditional Courses n = 1,176, with 3 institutions making up 48% of classes; Online Courses n = 
2,993, with 7 institutions making up 48% of classes). 


Table 2 
Frequency and Percentage of Highest Degree Awarded by Type of Course Instruction 


Traditional 2002-2008 2002-2008 
Courses Online Courses IDEA Database IDEA Online 


Highest Degree Awarded Frequency Percent Frequency Percent Frequency Percent Frequency Percent 


Associate's 1137 216 3,715 27.7 150,962 214 7,989 15.1 
Baccalaureate 1,084 20.6 611 4.5 104,743 14.9 8,869 16.7 
First professional degree 14 3 0 0.0 1,959 0.3 14 .03 
Master's 16944 321 4423 330 192,049 273 16982 32.0 
papal 49 9 1,010 75 43302 641 5,175 9.8 
Doctorate 1214 23.0 3,657 273 211389 300 13,891 262 
Not applicable 80 15 0 0.0 183 03 80 2 
Total 5272 100.0 13,416 100.0 704,587 100.0 53,000 ‘100.0 


Before comparing the student ratings, it was important to examine whether similar types 
of students were enrolled in traditional and online courses. Table 3 presents the frequency and 
percentage of principal types of students enrolled across type of course, as reported by the 
instructor on the Faculty Information Form. The percentage of students enrolled in lower 
division, general education classes was very similar in traditional (25.5%) and online (24.4%) 
courses. Slight differences between course modalities were found, however, in lower division, 
specialized; upper division, specialized; and graduate/professional classes. The largest difference 
was observed in the percentage of students enrolled in graduate/professional classes, which was 
somewhat greater for online (26.1%) than traditional (15.3%) courses. Although none of these 
differences is large, they should be kept in mind when interpreting the results of this report. 
Table 3 also shows that student response rates to IDEA Online were highest in 
graduate/professional classes and lowest in lower division classes, regardless of course modality. 


Table 3 
Frequency and Percentage of Principal Type of Student Enrolled by Type of Course Instruction 
Traditional Courses 


Response Rate 


Student Type Frequency Percent M SD 
Lower Division, General Education 1,228 25.5% 0.54 0.25 
Lower Division, Specialized 1,073 22.3% 0.56 0.25 
Upper Division, General Education 244 5.1% 0.60 0.23 
Upper Division, Specialized 1,076 22.4% 0.62 0.24 
Graduate/Professional 734 15.3% 0.74 0.26 
Combination 453 9.4% 0.55 0.24 
Total 4,808 100.0% 0.60! 0.26 
Missing 464 8.8% - - 


Online Courses 
Response Rate 


Student Type Frequency Percent M SD 
Lower Division, General Education 2,983 24.4% 0.40 0.20 
Lower Division, Specialized 1,924 15.7% 0.43 0.21 
Upper Division, General Education 752 6.1% 0.49 0.20 
Upper Division, Specialized 1,684 13.8% 0.54 0.22 
Graduate/Professional 3,199 26.1% 0.61 0.22 
Combination 1,698 13.9% 0.47 0.22 
Total 12,240 100.0% 0.50 0.23 
Missing 1,176 8.8% - - 


Note: M = mean; SD = standard deviation. 


The percentage of experienced and novice campus users of the IDEA Student Ratings 
system in both types of courses was computed (see Table 4). The vast majority of instructors 
were experienced users of IDEA in both traditional (62.8%) and online courses (73.7%). As one 
might expect, the mean student response rates were slightly higher for experienced users; this 
was especially true in traditional courses. 


Table 4 
Descriptive Statistics for Response Rate by User Status and Type of Course Instruction 
Traditional Courses Online Courses 
Response Response 
User Status N % Rate User Status N % Rate 
M SD M SD 


Experienced 3,313 62.8% 0.61 0.25 | Experienced 9,881 73.7% 0.50 0.23 
Novice 1959 37.2% 0.53 0.28 Novice 3,535 26.3% 0.47 0.23 
Total 5,272 100.0% 0.58' 0.26 Total 13,416 100.0% 0.50 0.23 

Note: M = mean; SD = standard deviation. 
'The response rate for traditional courses is different in Tables 3 and 4 because of the 464 classes 
excluded in the first analysis. 


Instrumentation 


Faculty Information Form (FIF). The FIF solicits information about each course from the 
instructor. Each campus determines the start and end dates for the survey completion. The online 
version is delivered to faculty via e-mail. Instructors rate each of 12 learning objectives as either 
3 = “Essential,” 2 = “Important,” or 1 = of “Minor or No Importance.” Instructors respond to 
contextual questions about the primary and secondary instructional approaches to the course 
(e.g., lecture, discussion/recitation, seminar); course requirements (e.g., writing, oral 
communication, group work); whether any of several factors may have had a positive, negative, 
or neutral impact on students’ learning (e.g., physical facilities, student enthusiasm to take the 
course, technical/instructional support); and the primary type of student enrolled (e.g., first 
year/sophomore meeting general education requirements, upperclassmen non-majors, graduate or 
professional students). They also indicate whether the course was team taught and whether it was 
taught through distance learning (see Appendix A).” 


Student Ratings Forms. The IDEA Center recommends students complete ratings at least 
after the first half of the course but not the last day of class. On the 47-item IDEA Diagnostic 
Form, students are asked to indicate how frequently their instructor used each of 20 teaching 
methods, using a scale of 1 = “Hardly Ever,” 2 = “Occasionally,” 3 = “Sometimes,” 4 = 
“Frequently,” and 5 = “Almost Always.” Students also rate their progress on each of the same 12 
learning objectives their instructor rated for importance. Students respond with | = “No apparent 
progress,” 2 = “Slight progress,” 3 = “Moderate progress,” 4 = “Substantial progress,” and 5 = 
“Exceptional progress.” Additional questions concern course characteristics, the student’s 
characteristics (i.e., work habits, motivation), the student’s overall rating of course and instructor 
excellence, and four additional experimental items addressing teaching methods. The 18-item 
IDEA Short Form includes the 12 learning objectives, three summary measures of teaching 
effectiveness, two items addressing student characteristics (i.e., student motivation and work 
habits), and one experimental item related to student background (see Appendix A). 


Four survey delivery methods are available online: survey links available through a 
Blackboarde Building Block, e-mail, the course website, or a combination of all three. Students 
completing the online survey are restricted to one submission. 


RESULTS 


Because the samples for this research are so large and measures of statistical significance 
are sensitive to large sample size, comparisons between paper and online survey administration 
were focused primarily on “practical significance” (i.e., are differences meaningful enough to 
change the interpretation of results) and an examination of results to determine if consistently 
different patterns emerged. 


Are student response rates to IDEA similar between traditional and online courses? 


On average, the proportion of students responding to the paper version of IDEA is higher 
than the online version (Ms = .78 vs. .55, respectively; see Benton, Webster, Pallett, and Gross, 


° The question on the FIF reads “Is this class taught through distance learning?” along with the response options of 
“Yes” or “No.” However, the question offers no possibility for distinguishing between courses taught online versus 
off-campus. In addition, the question is optional and is not always completed by the instructor. Therefore, this item 
could not be used to identify online courses. 
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2010). Accordingly, the possible difference in response rate between traditional and online 
courses was assessed. As shown in Table 5, traditional courses (M = .58, SD = .26) in this sample 
had higher response rates than did online courses (M = .50, SD = .23); the magnitude of this 
difference was about one-third standard deviation. As can be seen in Table 6, the overall mean 
student response rate for online survey delivery declined from a high of 56% in 2002 to 51% in 
2008. The general decline has been somewhat more dramatic in online courses. 


Table 5 
Means and Standard Deviations for Proportion of 
Students Responding by Type of Course Instruction 


Course 

Instruction a SY 
Traditional 58 26 5,272 
Online 50 ~~ .23 13,416 
Total 52 24 18,688 


Note: M= mean; SD = standard deviation. 


Table 6 
Means and Standard Deviations for Proportion of Students Responding by Year and Type 


of Course Instruction 


Overall Traditional Courses Online Courses 
Year M SD N M SD N M SD N 
2002 56 19 231 .60 18 15 56 19 216 
2003 56 19 326 .63 14 30 55 .20 296 
2004 50 .23 466 43 23 109 52 23 357 
2005 53 23 1,105 51 27 355 54 21 750 
2006 54 23 2,686 .64 25 754 50 22 1,932 
2007 52 23 4,536 62 25 1,032 49 23 3,504 
2008 52 24 9,338 57 727 2,977 49 23 6,361 
Total 52 24 18,688 58 26 5,272 50 23 13,416 


The cause of the moderate difference in response rates between online and traditional 
courses becomes clearer when examining Tables 7 and 8. Overall, traditional classes that used 
the Diagnostic Form exhibited slightly higher response rates compared to those using the Short 
Form (see Table 7). Although this was not the case every year (as indicated in Table 8), the 
differences between form types in 2007 and 2008 (boxed area) weighed more heavily because 
classes in those years made up the preponderance of this sample of traditional courses (76%). 
Meanwhile, response rates for online courses did not vary as much between the Diagnostic and 
Short Forms in 2007 and 2008. 


Table 7 
Means and Standard Deviations for Proportion of Students Responding by Type of 
Course Instruction and Form Type 


Overall Traditional Courses Online Courses 
Form Type M SD N M SD N M SD N 
Short 51.24 4,149 53 .26 1,566 A9 23 2,583 
Diagnostic 53 24 14,539 61 26 3,706 50 23 10,833 
Total 52 .24 18,688 58 .26 5,272 50 23 13,416 


Note: M = mean; SD = standard deviation. 


Table 8 


Means and Standard Deviations for Proportion of Students Responding Disaggregated by Type 
of Course Instruction, Year, and Form Type 


Traditional Courses 


Form Type/Statistic 


oer Short Diagnostic Total Short Diagnostic Total Short Diagnostic Total 
M M M SD SD SD N N N 
2002 59 61 .60 24 ab5 18 5 10 15 
2003 62 .63 .63 .16 14 14 13 17 30 
2004 53 Al 43 .20 pup Ls. 18 91 109 
2005 ayo 50 == 8 Be vos wall 54 301 355 
2006 .66 .64 .64 Da 220 25 100 654 754 
2007 56 65 .62 24 a25 sas) 328 704 1,032 
2008 7 ES .26 oD) 1,048 1,929 2977 
Total 3 61 58 .26 .26 .26 1,566 3,706 S202 
Online Courses 
Form Type/Statistic 
ee Short Diagnostic Total Short Diagnostic Total Short Diagnostic Total 
M M M SD SD SD N N N 
2002 Jl 59 56 a9, 19 19 80 136 216 
2003 4 56 Fs) a 21 20 100 196 296 
2004 46 61 a2 | ie 23 220 137 Bo 
2005 46 59 4 td .20 21 213 477 750 
2006 48 50 50 a 2a ae 350 1,582 1,932 
2007 50 A8 49 2D Pa) 23 495 3,009 3,504 
2008 49 A9 49 2 ve 5 1,065 5,296 6,361 
Total 49 50 50 ve Dee: 225 2,009 10,833 13,416 


Note: M = mean; SD = standard deviation. 


Additional explanations may suggest why response rates were somewhat higher in 
traditional courses. Perhaps because of face-to-face contact with students, an instructor in a 
traditional course has more influence on them. Some traditional instructors also may have access 
to computer labs where students can complete the ratings in-class. Others may work in 
institutions where students are encouraged to bring laptops to class. In contrast, most online 
instructors never meet students in person, which may diminish the instructor’s influence on 


student compliance. 


Also of interest was whether student response rates varied by size of class. Class sizes 


were categorized into subgroups separately by type of course instruction (see Table 9). For 


online courses, the highest student response rate was found in classes enrolling fewer than 10 


students (58%). Response rates declined as enrollments increased. In traditional courses, the 
response rates for class sizes less than 10 (64%) and greater than 39 (63%) were about the same. 


Response rates were relatively lower, regardless of course type, in classes enrolling anywhere 


from 10 to 39 students. The lowest response rate (41%) was found in online classes with 
enrollments exceeding 39 students. 


Table 9 

Means and Standard Deviations for Proportion of Students 
Responding Disaggregated By Type of Course Instruction 
and Number of Students Enrolled 


Traditional Courses Online Courses 
Students 
Enrolled M SD N M SD N 
<10 64 .26 879 58 25 2,403 
10-14 8° <5 1,083 soe «22. “2433 
15-24 34 = 25 2,002 A8 22 5,387 
25-39 59 27 811 A5 21 2,366 
> 39 63 .28 497 Al 21 827 
Total 58 26 S272 50 23 13,416 


Note: M = mean; SD = standard deviation. 
M (and SD) students enrolled for traditional = 21.97 (22.36), online = 20.17 (18.61). 


Correlations between response rate and student ratings. Because students’ response rate 
was slightly higher in traditional courses, we computed correlations between response rates and 
student ratings (Diagnostic Form) separately for both types of courses. As indicated in Table 10 
(below and continued on the next page), although the correlations between response rate and 
student ratings were slightly higher in online courses, the correlations were quite low in both 
types of courses. Thus, in this sample response rate did not have a strong relationship with 
student ratings in either type of course. 


Table 10 
Correlations between Student Ratings and Response Rate by Type of Course Instruction 
erent Traditional | Online 
Courses | Courses 

TM |. Displayed a personal interest in students and their learning .08 .16 
TM 2. Found ways to help students answer their own questions .04 14 
TM 3. Scheduled course work (class activities, tests, projects) in ways which encouraged 

students to stay up-to-date in their work -.01 10 
TM 4. Demonstrated importance and significance of the subject matter .05 16 
TM 5. Formed “teams” or “discussion groups” to facilitate learning 12 19 
TM 6. Made it clear how each topic fit into the course .04 .16 
TM 7. Explained the reasons for criticisms of student academic performance .06 15 
TM 8. Stimulated students to intellectual effort beyond that required by most courses .04 15 
T 9. Encouraged students to use multiple resources (e.g., data banks, library holdings, 

outside experts) to improve understanding -.01 14 
TM 10. Explained course material clearly and concisely .00 .10 
TM 11. Related course material to real life situations .09 19 
TM 12. Gave projects, tests, etc. which covered the most important parts of the course -.06 .08 
TM 13. Introduced stimulating ideas about the subject .05 16 
TM 14. Involved students in “hands on” projects such as research, case studies, or “real 

life” activities 07 21 
TM 15. Inspired students to set and achieve goals which really challenged them .04 17 
TM 16. Asked students to share ideas and experiences with others whose backgrounds and 

viewpoints differ from their own .05 19 
TM 17. Provided timely and frequent feedback on tests, reports, projects, etc. to help 

students improve -.01 07 
TM 18. Asked students to help each other understand ideas or concepts .08 18 
TM 19. Gave projects, tests, or assignments that required original or creative thinking O01 .16 
TM 20. Encouraged student-faculty interaction outside of class (office visits, phone calls, 

e-mail, etc.) .05 15 
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Table 10 (continued) 


Correlations between Student Ratings and Response Rate by Type of Course Instruction 


Sanden: Traditional | Online 
Courses _| Courses 

TM 44. The instructor used a variety of methods—not only tests—to evaluate student 

progress on course objectives. 02 18 
TM 45. The instructor expected students to take their share of responsibility for learning. -.07 12 
TM 46. The instructor had high achievement standards in this class. .00 .16 
TM 47. The instructor used educational technology (e.g., Internet, email, computer 

exercises, multi-media presentations, etc.) to promote learning -.01 13 
Obj |. Gaining factual knowledge (terminology, classifications, methods, trends) -.01 .09 
Obj 2. Learning fundamental principles, generalizations, or theories -.02 .08 
Obj 3. Learning to apply course material (to improve thinking, problem solving, and 

decisions) .O1 12 
Obj 4. Developing specific skills, competencies, and points of view needed by 

professionals in the field most closely related to this course O01 13 
Obj 5. Acquired skills in working with others as a member of a team 07 15 
Obj 6. Developing creative capacities (writing, inventing, designing, performing in art, 

music, drama, etc.) -.02 10 
Obj 7. Gaining a broader understanding and appreciation of intellectual/cultural activity 

(music, science, literature, etc.) -.04 .04 
Obj 8. Developing skill in expressing myself orally or in writing .00 12 
Obj 9. Learning how to find and use resources for answering questions or solving 

problems .04 .09 
Obj 10. Developing a clearer understanding of , and commitment to, personal values -.01 .09 
Obj 11. Learning to analyze and critically evaluate ideas, arguments, and points of view .O1 12 
Obj 12. Acquiring an interest in learning more by asking my own questions and seeking 

answers .00 ll 
CR 33. Amount of reading -.04 -.05 
CR 34. Amount of work in other (non-reading) assignments O01 .06 
CR 35. Difficulty of subject matter .04 .O1 
Self 36. I had a strong desire to take this course. .03 ll 
Self 37. I worked harder on this course than on most courses I have taken. .00 .08 
Self 38. I really wanted to take a course from this instructor. .10 15 
Self 39. I really wanted to take this course regardless of who taught it. -.01 .O1 
Self 43. As arule, I put forth more effort than other students on academic work. -.04 .06 
GL 40. As a result of taking this course, I have more positive feelings toward this field of 

study. 01 12 
GL 41. Overall, I rate this instructor as an excellent teacher. 0.00 0.10 
GL 42. Overall, I rate this course as excellent. -0.03 0.09 
PRO. Progress on Relevant (Important and Essential) Objectives 0.03 0.12 
PROadj. Adjusted Progress on Relevant Objectives Score 0.05 0.08 


‘Copyright © IDEA Center 1998 


Note: Ns for traditional and online courses ranged from 3,704 to 5,272 and 10,784 to 13,416, 


respectively. 
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Are instructors’ ratings of the importance of the 12 IDEA learning objectives similar 
between types of courses? 


Table 11 presents descriptive statistics for instructor ratings of importance on each of the 
12 IDEA learning objectives by type of course instruction and for the overall IDEA database. 
The “% Total” columns in Table 11 indicate the percentage of instructors rating an objective 
either “important” or “essential.” Those percentages did not differ meaningfully between 
traditional and online courses. Differences in the “% Total” ranged from |0.9%| for 
“communication skills” (Objective 8) to |11.3%| for “team skills” (Objective 5). Similarly, when 
examining mean differences in importance ratings of the 12 learning outcomes, differences 
ranged from |.01| for “communication skills” to |.17| for “creative capacities (Objective 6). Given 
that the average SD for the objectives was quite large (.74), mean ratings of importance did not 
differ meaningfully between online and traditional courses. 
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Table 11 
Frequencies and Descriptive Statistics for Instructor (FIF) Ratings of Learning Objectives 


Traditional Courses Online Courses IDEA Database 2002-2008 
Learning Outcome %I KE ae M sD es %I ME Bee M SD he %I WE ae M SD ae 
FR 1. Factual knowledge 33.6 44.1 77.7 2.22 0.79 4,753 | 33.0 48.5 81.5 2.30 0.76 12,553 | 30.4 49.3. 79.7 2.29 .78 647,582 
FR2. Principles and theories | 33.5 41.7 75.2. 217 0.80 4,725 | 338 43.1 769 2.20 0.79 12,501 | 33.9 41.9 75.8 218 .79 642,218 
FR3. Applications 36.8 45.4 82.2 227 0.75 4,743 | 35.9 48.0 83.9 232 0.74 12,554|38.2 39.5 77.7 217 .77 643,623 
ERS. Professional skills, 32.0 33.2 65.2 1.99 082 4,698 | 31.2 30.5 61.7. 1.92 0.83 12,409] 30.0 29.3 59.3. 1.89 .83 625,806 
viewpoints 
FRS. Team skills 26.0 111 37.1 148 069 4,628/193 65 258 1.32 0.59 12,211]22.7 89 316 1.40 .65 612,520 
FR6. Creative capacities | 20.1 13.2 333 147 0.72 4,589] 15.7. 7.1 228 130 0.59 12,168] 14.9 11.5 264 138 .68 609,445 
FR7. Broad liberal education | 18.4 13.1 31.5 145 0.71 4,600] 133 10.9 24.2 135 0.67 12,199] 164 10.6 27.0 1.38 .67 609,518 
FR8. Communication skills | 29.8 20.0 49.8 1.70 0.78 4,671 | 318 18.9 50.7 1.9 0.77. 12,262|263 19.5 45.8 1.65 .79 620,235 
FR9. Find, use resources | 35.6 18.6 54.2 1.73 0.76 4,663 | 37.7 22.0 59.7 182 0.77 12326/304 123 42.7 1.55 .70 613,826 
FRO, Values development | 218 9.1 30.9 1.40 0.65 4,589] 201 81 282 136 0.63 12,160/17.1 7.1 24.2 131 60 603,964 
FR11. Critical analysis 293 25.1 54.4 1.80 0.82 4,653 | 32.0 27.0 59.0 1.86 0.81 12,346]27.8 208 486 1.69 .79 620,827 
FR12, Interest in learning | 36.2 17.5 53.7 1.71 0.75 4,608 | 335 13.9 474 1.61 0.72 12,205] 30.5 12.2 42.7 1.55 .70 607,185 


Note: Note. M= mean; SD = standard deviation. % I = % Important; % E = % Essential. 


M number of objectives selected as “important” or “essential” for traditional and online courses = 5.73 (SD = 3.20) and = 5.74 (SD = 3.07), 


respectively. For the overall 2002-2008 IDEA Database, M = 5.27 (SD = 2.89). 
Instructors rated importance of learning objectives as 1 = Minor or No Importance, 2 = Important, or 3 = Essential. 
Valid N = Number of responses from all classes excluding missing responses. 


13 


Are the inter-correlations among the instructor ratings of the 12 learning objectives similar 
across type of course? 


Table 12 presents inter-correlations among instructor ratings of the importance of the 12 
learning objectives, computed separately for traditional and online courses. The only 
conspicuous difference was that the correlation between Objective 6 (Developing creative 
capacities) and Objective 8 (Developing skill in expressing oneself orally or in writing) was 
higher in online (r = .47) than traditional (r = .29) courses. However, in both cases the 
correlation was moderately positive. Given the high number of correlations computed, this single 
difference was not considered meaningful. In general, then, there were no systematic differences 
between traditional and online courses in the correlations among instructor ratings of objectives. 


Table 12 
Inter-Correlations of IDEA Faculty Information Form Faculty Ratings (FR) by Type of Course 
Instruction 


Traditional 


Item FRI | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRS | FR1I0 FRI1 


FR1 1 


FR2 0.42 1 
FR3 0.08 | 0.22 1 


FR4 0.08 | 0.07 | 0.29 1 


FR5 -0.07 | 0.01 | 0.20 | 0.21 1 


FR6 -0.11 | -0.04 | 0.13 | 0.21 | 0.29 1 


FR7 0.01 | 0.04 | 0.00 | -0.03 | 0.14 | 0.37 1 


FR8 -0.11 | -0.03 | 0.12 | 0.05 | 0.34 | 0.29 | 0.27 1 


FRI 0.08 | 0.14 | 0.32 | 0.24 | 0.32 | 0.21 | 0.14 | 0.39 1 


FR10 | -0.03 | 0.06 | 0.16 | 0.08 | 0.35 | 0.24 | 0.28 | 0.33 | 0.31 1 


FR11 | -0.03 | 0.15 | 0.24 | 0.03 | 0.24 | 0.19 | 0.29 | 042 | 0.38 | 0.40 1 


FR12 | 0.09 | 0.18 | 0.29 | 0.19 | 0.33 | 0.29 | 0.32 | 0.38 | 0.52 | 0.43 0.50 


Online 


Item FRI | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRY | FR10 | FRI1 


FR1 1 


FR2 0.44 1 
FR3 0.03 | 0.22 1 


FR4 0.00 | 0.06 | 0.32 1 


FR5 -0.05 | 0.07 | 0.22 | 0.26 1 


FR6 -0.01 | 0.07 | 0.15 | 0.16 | 0.31 1 


FR7 0.08 | 0.09 | 0.01 | 0.00 | 0.17 | 0.41 1 


FR8 -0.06 | 0.02 | 0.12 | 0.09 | 0.30 | 0.47 | 0.33 1 


FRI 0.08 | 0.10 | 0.27 | 0.28 | 0.29 | 0.29 | 0.19 | 0.38 1 


FR10 | 0.06 | 0.15 | 0.20 | 0.18 | 0.34 | 0.34 | 0.29 | 0.35 | 0.32 1 


FR11 | 0.00 | 0.16 | 0.27 | 0.08 | 0.27 | 0.33 | 0.29 | 0.47 | 0.38 | 0.39 1 
FR12 | 0.16 | 0.24 | 0.27 | 0.22 | 0.35 | 0.37 | 0.34 | 0.43 | 0.50 | 0.47 | 0.52 


Note: Ns for Traditional and Online Courses = 4,589 to 4,753 and = 12,074 to 12,553, respectively. See 
Table 11 for item descriptions. 
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Are there differences in students’ ratings of progress on the 12 learning objectives? 


Table 13 presents student ratings of individual items on the IDEA Diagnostic Form by 
type of course instruction. For each item, the magnitude of the difference between traditional and 
online courses is noted, as well as the approximate value of d, the standardized mean difference 
(Cohen, 1988). A measure of effect size, d = [(Traditional Mean — Online Mean) / pooled 
standard deviation].* Cohen (1988) considered effect sizes approximating .20 (1/5 standard 
deviation) as small, .50 as medium, and .80 as large. The effect sizes in Table 13 indicate 
students’ self-reported progress on the 12 objectives (“Obj 1” to “Obj 12”) was very similar 
across the two types of courses. Therefore, students in the current sample reported similar 
progress regardless of whether they were enrolled in on-campus or online courses. 


One of the important hallmarks of IDEA student ratings is that students consistently 
report making greater progress on objectives their instructor rated as important or essential 
(Hoyt, 1973; Hoyt & Lee, 2002a). Table 14 shows that this was the case, regardless of type of 
instruction. Across both traditional and online courses, students consistently reported greater 
progress on important and essential objectives. This provides evidence of criterion-related 
validity for IDEA student ratings in both traditional and online courses. 


Do students’ ratings of how frequently the instructor used various teaching methods differ 
between traditional and online courses? 


Student ratings of the frequency of 20 teaching methods (TM 1| to TM 20) were highly 
similar across the type of course (see Table 13). Students in traditional and online courses did not 
differ meaningfully in their ratings of how frequently their instructor used various teaching 
methods. This implies that instructors are perceived to employ similar teaching methods across 
course modalities, which supports the generalizability of the 20 teaching methods. However, one 
experimental teaching method item did differ: TM 47, “The instructor used educational 
technology (e.g., Internet, e-mail, computer exercises, multi-media presentations, etc.) to 
promote learning.” As indicated in Table 13 (see TM 47), online students (M = 4.44) rated their 
instructors higher on this method than did traditional students (M = 4.16). The d of -.49 indicates 
a medium effect size due to course modality. This difference makes sense when one considers 
that, by their very nature, online courses rely heavily upon technology. 


On two other items, small effect sizes were found between online and traditional courses. 
First, students in online courses (M = 3.48) reported a greater amount of reading (see TM 33) 
than did those in traditional courses (M = 3.23). This is to be expected given how much 
information is typically presented in text form in online classes; students are often expected to 
read lectures, e-mail, and instructor and student postings. In contrast, traditional courses 
frequently rely upon in-class lectures to deliver content, which requires more listening than 
reading. Second, students in online courses (M = 3.37) reported less motivation for taking the 
course from “this instructor” (see TM 38) than did those in traditional courses (M = 3.59). It is 
likely that students in online programs are less likely to know an instructor until they take the 
course. Therefore, they are less likely to have strong preferences for a specific instructor. 


: s= V (n; — 1s; + (ny — 1)s." 
n)+ Nh- 2 
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Table 13 


Student Ratings of Individual Items on the IDEA Diagnostic Form by Type of Course Instruction 


ee Re anne 2002-2008 (Traditional) 2002-2008 (Online) Traditional Pate 
M SD M SD | ABSA M SD | 42441 ———— 

T™ 1 4.43 0.49 433 | 0.59 | 0.10 4.19 | 068 | 0.24 0.14 0.21 
TM 2 4.23 0.53 417 | 0.63 | 0.06 407 | 068 | 0.16 0.10 0.15 
TM 3 4.30 0.50 424 | 061 | 0.06 am 0.60 | -0.02 -0.08 -0.13 
T™M 4 44] 0.47 432 | 0.58 | 0.09 Woo 0.63 | 0.18 0.09 0.15 
T™ 5 3.68 0.96 3741 094 | -0.06 |eaee| 1.02 | 0.02 0.08 0.08 
TM 6 4.30 0.52 422 | 0.62 | 0.08 412 | 068 | 0.18 0.10 0.15 
T™M 7 3.99 0.60 3.98 | 071 | 0.01 0.79 | (0.12 0.11 0.14 
T™M 8 4.05 0.58 403 | 0.68 | 0.02 4.00 | 0.69 | 0.05 0.03 0.04 
T™ 9 3.98 0.67 aoe 0.69 | -0.04 [a 0.71 | -0.13 -0.09 -0.13 
T™M 10 4.24 0.61 415 | 0.71 | 0.09 409 | 0.73 | 0.15 0.06 0.08 
TM 11 431 0.58 427 | 0.63 | 0.04 415 | 0.71 0.16 0.12 0.17 
TM 12 4.35 0.51 426 | 0.61 | 0.09 435 | 056 | 0.00 -0.09 -0.16 
T™ 13 4.17 0.58 415 | 0.66 | 0.02 4.06 | 0.71 0.11 0.09 0.13 
T™ 14 3.93 0.80 4.02 | 0.76 | -0.09 | 3.97 | 0.83 | -0.04 0.05 0.06 
T™ 15 3.97 0.63 3.99 | 069 | -0.02 | 3.95 | 072 | 0.02 0.04 0.06 
T™ 16 3.87 0.78 389 | 081 | -002 | 3.93 | 088 | -0.06 -0.04 -0.05 
TM 17 4.23 0.60 een 0.73 | 0.11 407 | 080 | 0.16 0.05 0.06 
T™M 18 3.96 0.65 397 | 071 | -0.01 | 386 | 085 | 0.10 0.11 0.14 
T™ 19 4.07 0.64 410 | 0.67 | -0.03 | 4.16 | 0.68 | -0.09 -0.06 -0.09 
T™ 20 4.07 0.62 4.08 | 0.69 | -0.01 | 3.98 | 0.77 | 0.09 0.10 0.13 
TM 44 3.94 0.60 397 | 064 | -0.03 | 407 | 068 | -0.13 -0.10 -0.15 
TM 45 4.35 0.36 441 | 043 | -0.06 | 452 | 041 | -0.17 -0.11 0.26 
TM 46 4.19 0.44 422 | 052 | -0.03 | 427 | 0.51 | -0.08 -0.05 -0.10 
TM 47 3.95 0.72 416 | 0.65 | -021 | 444 | 0.54 | -0.49 -0.28 -0.49 
Obj 1 4.14 0.50 4.09 | 0.58 | 0.05 4.09 | 054 | 0.05 0.00 0 
Obj 2 4.09 0.51 4.06 | 0.58 | 0.03 4.05 | 055 | 0.04 0.01 0.02 
Obj 3 4.12 0.52 4.08 | 0.60 | 0.04 4.08 | 058 | 0.04 0.00 0 
Obj 4 4.07 0.54 404 | 061 | 0.03 4.02 | 0.60 | 0.05 0.02 0.03 
Obj 5 3.59 0.79 360 | 080 | -001 | 344 | 082 | 0.15 0.16 0.20 
Obj 6 3.59 0.77 3.66 | 0.76 | -0.07 | 3.64 | 0.75 | -0.05 0.02 0.03 
Obj 7 3.58 0.74 on 0.77 | -0.04 eee 0.77 | 0.04 0.08 0.10 
Obj 8 3.60 0.77 3.56 | 0.78 | 0.04 3.68 | 0.76 | -0.08 ae -0.16 
Obj 9 3.80 0.61 3.80 | 0.66 | 0.00 om 0.63 | -0.13 “053 -0.20 
Obj 10 3.66 0.70 3.65 | 0.75 | 0.01 371 | 0.72 | -0.05 0.06 -0.08 
Obj 11 3.82 0.64 fom 0.70 | 0.03 3.88 | 0.67 | -0.06 -0.09 -0.13 
Obj 12 3.91 0.58 3.86 | 0.67 | 0.05 3.89 | 065 | 0.02 -0.03 -0.05 
CR 33 A) 0.74 323. | 079 | -0.01 | 348 | 059 | -0.26 -0.25 -0.38 
CR 34 3.49 0.58 353 | 062 | -0.04 | 3.53 | 054 | -0.04 0.00 0 
CR 35 3.46 0.58 349 | 063 | -0.03 | 342 | 055 | 0.04 0.07 0.12 
Self 36 Be 0.70 ome 0.71 | -0.12 Meee 071 | -0.04 0.08 0.11 
Self 37 3.67 0.57 Sym 0.62 | -0.05 [Mem 0.57 | -0.05 0.00 0 
Self 38 3.56 0.71 oom 0.77 | -0.03 Me 0.73 | 0.19 0.22 0.30 
Self 39 3.53 0.61 Soi 0.68 | -0.08 [Mies 0.66 | -0.10 -0.02 -0.03 
Self 43 3.80 0.39 385 | 042 | -0.05 | 386 | 044 | -0.06 -0.01 -0.02 
GL 40 4.01 0.60 3.99 | 068 | 0.02 3.97 | 067 | 0.04 0.02 0.03 
GL 41 4.29 0.61 420 | 0.72 | 0.09 4.18 | 0.71 0.11 0.02 0.03 
GL 42 4.07 0.61 4.05 | 0.69 | 0.02 4.06 | 068 | 0.01 -0.01 -0.01 
PRO on 8.74 (emo 9.92 | 1.17 eee 9.72 1.10 -0.07 -0.01 
PROadj | 51.01 898 | 48.91 | 1037 | 2.10 | 4843 | 1060 | 2.58 0.48 0.05 


Note: TM = Teaching Method; Obj = Teaching Objective; CR = Course Rating; Self = Self-Rating; GL = Global; PRO = Progress on relative 
objectives; adj = adjusted. ABS A = Absolute value of 2002-2008 IDEA Database mean minus 2002-2008 Traditional or Online mean. Approx 
d=measure of effect size (see page 15 footnote). See Table 10 for item descriptions. 
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Table 14 
Descriptive Statistics for Student Ratings of Progress on Objectives by Type of Course 
Instruction at Each Level of Instructor Rating of Importance 


Traditional Courses 


Minor or No 


Important & 


Importance Important Essential Essential 

Learning Outcome M SD N M SD N M SD N M_ SD N 

1. Factual knowledge 3.95 0.62 1,063 | 4.09 0.54 1,595] 4.17 0.54 2,095 | 4.14 .54 3,690 
2. Principles and theories 3.94 0.62 1,175 | 4.10 0.54 1,582 | 4.11 0.54 1,968} 4.11 .54 3,550 
3. Applications 4.00 0.57 847 | 4.04 0.61 1,745 | 4.14 0.57 2,151 | 4.10 .59 3,896 
4. Professional skills, viewpoints 3.93 0.59 1,632 | 4.04 0.61 1,504) 4.16 0.57 1,562] 4.10 59 3,066 
5. Team skills 3.43 0.79 2,911 | 3.82 0.72 1,204 | 4.06 0.59 513 3.89 68 1,717 
6. Creative capacities 3.55 0.75 3,060 | 3.83 0.69 924 | 4.04 0.67 605 | 3.91 .68 = 1,529 
7. Broad liberal education 3.50 0.76 3,149 | 3.79 0.73 848 | 4.02 0.67 603 | 3.89 .71 1,451 
8. Communication skills 3.35 0.79 2,342 | 3.64 0.72 1,394 | 400 0.60 935 | 3.78 .67 2,329 
9. Find, use resources 3.70 0.65 2,136 | 3.86 0.64 1,660 | 3.94 0.63 867 | 3.89 64 2,527 
10. Values development 3.56 0.74 3,169 | 3.80 0.71 1,002 | 3.98 064 418 | 3,85 .69 1,420 
11. Critical analysis 3.65 0.70 2,118 | 3.85 0.68 1,365 | 3.96 0.63 1,170 | 3.90 .66 2,535 
12. Interest in learning 3.81 0.65 2,135 | 3.89 0.65 1,668 | 3.93 0.68 805 | 3.90 .66 2,473 

Online Courses 
Minor or No Important & 
Importance Important Essential Essential 

Learning Outcome M SD N M_ SD N M_ SD N M SD N 

1. Factual knowledge 4.06 0.53 2,329 | 4.08 0.53 4,139] 4.11 0.54 6,085 | 4.10 54 10,224 
2. Principles and theories 4.04 0.55 2,881 | 4.05 0.54 4,226 | 4.07 0.54 5,394] 4.06 54 9,620 
3. Applications 4.03 0.58 2,025 | 4.07 0.56 4,506 | 4.12 0.57 6,023 | 4.10 .57 10,529 
4. Professional skills, viewpoints 3.94 0.57 4,755 | 4.03 0.59 3,866 | 4.14 0.58 3,788 | 4.08 59 7,654 
5. Team skills 3.33 0.81 9,056 | 3.72 0.72 2,360 | 3.82 0.77 795 | 3.75.73 3,155 
6. Creative capacities 3.60 0.75 9,391 | 3.76 0.72 1,907 | 3.92 0.68 870 | 3.81 .71 = 2,777 
7. Broad liberal education 3.48 0.75 9,245 | 3.64 0.75 1,625 | 3.91 0.72 1,329 | 3.76 .74 2,954 
8. Communication skills 3.52 0.78 6,054 | 3.78 0.69 3,896 | 3.96 0.64 2,312 | 3.85 .68 6,208 
9. Find, use resources 3.88 0.62 4,961 | 3.94 0.63 4,649 | 4.03 0.61 2,716] 3.98 62 7,365 
10. Values development 3.65 0.72 8,728 | 3.86 0.64 2,448 | 3.93 0.66 984 | 3.88 .65 3,432 
11. Critical analysis 3.78 0.69 5,063 | 3.91 0.65 3,947) 4.00 0.63 3,336 | 3.95 .64 7,283 
12. Interest in learning 3.88 0.64 6,411 | 3.89 0.64 4,093 | 3.93 0.65 1,701 | 3.90 .64 5,794 


Note: Students responded to all items on a scale of 1 = No Apparent Progress to 5 = Exceptional 


progress; I made outstanding gains on this objective. 


Are the correlations between instructors’ and students’ ratings of learning objectives 


similar in traditional and online environments? 


An indirect test of the validity of the IDEA ratings involves correlating students’ reported 
progress for each objective with the instructors’ ratings of the importance of those objectives. 
The highest correlations should be found in ratings of the same objectives (see Hoyt, 1973). The 
correlations in Table 15 confirm that correlations among ratings of the same objectives 
(indicated in bold along the diagonal) are, on average, higher in both traditional and online 
courses. The average correlation between instructor and student ratings of the same 12 learning 
outcomes was, however, somewhat higher in traditional (r = .19) than in online (r = .12) courses. 
Nonetheless, the average off-diagonal correlation was quite low in both traditional and online 
courses, rs = .03 and .01, respectively. This provides indirect evidence of the validity of the 
student ratings in both course modalities. 
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Table 15 
Correlations between Faculty Ratings and Student Ratings of Learning Objectives for 


Traditional and Online Course Instruction 


Traditional 
Item | FR1 | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRI FR10 FR11 FR12 
SR21 | .15 .09 .04 07 -.05 | -.07 | -.01 -.06 | -.03 -.04 -.04 -.01 
SR22 | .12 11 .05 .06 -05 | -.07 | -.01 -.05 | -.03 -.02 -.02 -.02 
SR23 | .01 02 .09 12 02 01 -.04 01 02 02 -.01 01 
SR24 | .02 01 .08 16 02 02 -.04 .00 .02 -.02 -.05 -.01 
SR25 | -.07 | -.05 .09 10 .29 .05 .00 12 07 10 .03 07 
SR26 | -.13 | -.11 02 11 12 .24 14 .20 07 .08 07 .09 
SR27 | -.04 | -.06 | -.04 | -.03 .03 15 25 15 01 10 12 11 
SR28 | -.13 | -.11 02 00 11 .09 .09 33 .10 14 17 .10 
SR29 |_ .01 -.01 .09 07 05 -.02 | -.05 13 15 .02 07 .05 
SR30 | -.06 | -.05 .03 .03 .08 .05 .05 14 .06 19 13 .10 
SR31 | -.05 | -.01 .06 .00 .04 .00 04 16 07 11 19 07 
SR32 | -.01 -.01 .06 .06 04 02 .03 09 .06 07 .08 .07 
Online 
Item FR1 | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRI FR10 FR11 FR12 
SR21 | .04 -.01 .00 05 -.02 | -.03 | -.04 | -.02 .00 -.01 -.04 .00 
SR22 | .02 .02 02 .05 -.01 -.02 | -.05 | -.01 -.01 01 -.02 01 
SR23 | -.06 | -.05 .06 12 .03 -.01 -.08 01 02 .03 -.02 .00 
SR24 | -.05 | -.06 05 14 .04 -.01 -.08 .00 .02 01 -.04 -.01 
SR25 | -.13 | -.10 07 12 .22 01 -.06 .05 04 .06 .03 01 
SR26 | -.15 | -.15 | -.01 07 07 13 .06 18 .06 .06 .08 .02 
SR27 | -.06 | -.09 | -.06 | -.03 .02 .09 18 13 .03 07 .08 .05 
SR28 | -.16 | -.13 | -.01 .03 .09 11 07 24 .06 11 14 05 
SR29 | -.05 | -.08 01 07 .03 01 -.03 07 .09 01 02 01 
SR30 | -.09 | -.08 01 .05 07 .03 .00 .09 .03 14 .06 .03 
SR31 | -.10 | -.07 .02 01 .06 .05 .03 13 02 .08 13 .04 
SR32 | -.08 | -.09 .02 07 .05 02 -.01 .06 .03 .05 .03 .03 


Average r off-diagonal, Traditional = .03, Online = .01. 
Ns for Traditional and Online Courses = 4,589 to 4,753 and = 12,160 to 12,554, respectively. 
See Table 10 for item descriptions. 
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In both types of courses, the average on-diagonal correlations were lower than those 
reported in Hoyt (1973) and Hoyt and Lee (2002a). However, those studies excluded from the 
analysis classes having fewer than 10 respondents, whereas in Table 15 all classes were included. 
In addition, Hoyt and Lee (2002a) excluded novice users and classes using the Short Form. 
Therefore the correlations were computed between instructor and student ratings on the 12 
learning outcomes three more times; first removing Short-Form users only (Table 16), then 
removing novice users only (Table 17), and then removing classes with < 10 respondents only 
(Table 18). In some cases, enacting the Technical Report 12 exclusion criteria slightly increased 
on-diagonal correlations. This was especially true when removing classes with fewer than 10 
respondents (see Table 18). 


Table 16 
Correlations between Faculty Ratings and Student Ratings of Learning Objectives for 
Traditional and Online Course Instruction (Excluding Short Form Users) 


Traditional 
Item | FR1 | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRY | FR10 | FR11 | FR12 
SR21 | .13 .08 .03 .07 -.07 | -.07 | -.03 | -.07 | -.02 -.04 -.04 -.02 
SR22 | .10 10 04 .07 | -.07 | -.06 | -.02 | -.06 | -.02 -.02 -.02 -.03 
SR23 | .00 01 .08 13 01 01 -.05 .02 03 .03 -.01 .00 
SR24 | .00 | -.0l 06 17 01 .03 -.05 | -.01 02 -.01 -.05 -.02 
SR25 | -.08 | -.08 .08 .09 .29 .O7 .00 13 .08 11 .05 .06 
SR26 | -.15 | -.13 01 12 13 .27 15 20 .06 10 .09 .09 
SR27 | -.08 | -.08 | -.05 | -.01 .04 19 .25 15 01 12 12 .13 
SR28 | -.15 | -.14 01 02 12 11 .08 33 11 16 18 11 
SR29 | -.02 | -.04 .08 09 .06 01 -.04 15 .16 .05 09 .05 
SR30 | -.08 | -.07 .03 .04 .08 .08 .06 16 .08 21 14 11 
SR31 | -.08 | -.03 .05 01 .04 02 .03 18 .08 13 .20 .08 
SR32 | -.04 | -.04 .05 .08 .05 .05 .03 ll .08 10 .09 .07 
Online 
Item | FR1 | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRI | FR10 | FR11 | FR12 
SR21 | .03 | -.01 .00 .06 | -.02 | -.04 | -.05 | -.04 | -.01 -.02 -.05 -.01 
SR22 | .02 .02 02 .06 | -.02 | -.03 | -.06 | -.03 | -.01 01 -.03 .00 
SR23 | -.07 | -.06 | .06 13 .03 -.02 | -.09 | -.01 02 .02 -.03 -.01 
SR24 | -.06 | -.07 .05 15 .03 -.03 | -.09 | -.01 01 01 -.05 -.02 
SR25 | -.15 | -.10 07 14 21 .00 | -.07 04 .03 .06 01 .00 
SR26 | -.16 | -.15 | -.01 .08 06 1 04 16 04 .06 .06 01 
SR27 | -.06 | -.09 | -.05 | -.01 01 .08 .16 11 .02 .07 .06 .04 
SR28 | -.17 | -.14 | -.01 .06 .08 09 .06 21 05 11 12 .04 
SR29 | -.06 | -.08 02 09 02 .00 | -.04 .06 .09 01 .00 .00 
SR30 | -.11 | -.09 01 .06 .06 02 -.01 .07 .02 13 04 01 
SR31 | -.11 | -.07 02 02 05 .03 02 11 .02 .08 1 .02 
SR32 | -.08 | -.09 02 .08 04 01 -.02 .05 .02 .05 .02 14 


Note: Average r on-diagonal, Traditional = .19, Online = .12. 

Average r off-diagonal, Traditional = .03, Online .004. 

Ns for Traditional and Online Courses = 3,292 to 3,396 and = 9,818 to 10,152, respectively. 
See Table 10 for item descriptions. 
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Table 17 


Correlations between Faculty Ratings and Student Ratings of Learning Objectives for 


Traditional and Online Course Instruction (Excluding Novice Users) 


Traditional 
Item | FR1 | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRY | FR10 | FR11 | FR12 
SR21 | .19 11 .03 .06 -.09 | -.11 .00 | -.08 | -.06 -.06 -.05 -.01 
SR22 | .16 14 04 .05 -09 | -.11 | -.02 | -.08 | -.06 -.04 -.03 -.02 
SR23 | .03 04 .09 12 01 -.01 | -.05 .00 01 02 -.02 01 
SR24| .04 .03 .08 .16 .03 01 -.04 | -.01 01 -.02 -.04 01 
SR25 | -.07 | -.05 09 .09 31 02 .00 12 .06 09 01 .O7 
SR26 | -.14 | -.12 01 10 11 .24 17 23 .O7 .07 10 10 
SR27 | -.03 | -.06 | -.06 | -.04 .00 12 .29 18 -.01 11 15 12 
SR28 | -.15 | -.13 | -.01 | -.03 10 .07 12 35 10 14 18 11 
SR29 | .00 | -.03 09 .O7 .06 | -.05 | -.05 14 15 01 .O7 04 
SR30 | -.05 | -.05 .03 -.01 .05 01 .08 15 .05 .20 15 1 
SR31 | -.04 | -.01 .04 | -.02 .02 | -.04 .05 15 .05 10 21 .06 
SR32 | .00 | -.01 .05 05 .03 -.02 .04 .08 .05 .08 .08 .08 
Online 
Item | FR1 | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRI | FR10 | FR11 | FR12 
SR21 |) .02 | -.01 .00 .O7 .0O | -.01 | -.02 | -.02 .00 -.02 -.05 .00 
SR22 | .01 .03 .03 .O7 O01 00 | -.04 | -.01 .00 .00 -.02 01 
SR23 | -.08 | -.05 .07 13 .05 02 | -.06 02 .02 .03 -.02 01 
SR24 | -.06 | -.06 .06 .16 .05 01 -.07 01 .03 01 -.04 .00 
SR25 | -.15 | -.10 07 14 .24 04 | -.05 .O7 .05 .05 .03 02 
SR26 | -.16 | -.15 .00 .08 09 14 .08 19 .O7 .06 .08 .03 
SR27 | -.08 | -.09 | -.06 | -.02 .03 11 19 14 .05 .06 .08 .06 
SR28 | -.16 | .-123 | .00 05 11 12 .09 25 .08 11 15 .06 
SR29 | -.07 | -.08 .03 09 .05 04 | -.01 09 11 02 .03 .03 
SR30 | -.10 | -.07 .03 .O7 .08 .05 01 1 04 15 .07 .05 
SR31 | -.11 | -.06 .03 .03 08 .O7 .03 14 .04 .08 13 05 
SR32 | -.09 | -.08 .03 09 .O7 05 01 .O7 .05 .05 .03 .04 


Note: Average r on-diagonal, Traditional = .20, Online = .13. 
Average r off-diagonal, Traditional = .03, Online = .02. 


Ns for Traditional and Online Courses = 2,943 to 3,095 and = 8,976 to 9,291, respectively. 


See Table 10 for item descriptions. 
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Table 18 
Correlations between Faculty Ratings and Student Ratings of Learning Objectives for 
Traditional and Online Course Instruction (Excluding Classes with < 10 Respondents) 


Traditional 
Item | FR1 | FR2 | FR3 | FR4 | FR5S | FR6 | FR7 | FR8 | FRO | FR10 | FR11 | FR12 
SR21 | .21 10 .04 .10 -.07 | -.13 | -.05 | -.09 | -.06 -.08 -.07 -.06 
SR22 | .17 14 .06 10 | -.06 | -.12 | -.05 | -.07 | -.06 -.04 -.03 -.05 
SR23 | .03 02 10 17 04 | -.03 | -.07 .02 .02 01 -.03 -.01 
SR24 | .05 03 10 23 04 | -.01 | -.09 .02 03 -.04 -.06 -.03 
SR25 | -.11 | -.07 13 15 35 .05 -.02 17 13 .07 05 .09 
SR26 | -.17 | -.16 05 16 17 .26 14 27 13 10 11 11 
SR27 | -.04 | -.09 | -.02 01 .04 14 28 19 .03 13 16 13 
SR28 | -.18 | -.16 .03 .05 17 13 12 38 14 16 .20 13 
SR29 | -.02 | -.03 13 14 ll .03 -.04 16 19 01 .O7 .06 
SR30 | -.07 | -.07 .06 .06 ll .05 .08 19 ll .26 17 13 
SR31 | -.07 | -.03 .06 .04 .08 01 .07 20 .08 15 23 10 
SR32 |} -.01 | -.03 08 11 .08 02 04 13 .09 .09 1 .09 
Online 
Item | FR1 | FR2 | FR3 | FR4 | FR5 | FR6 | FR7 | FR8 | FRI | FR10 | FR11 | FR12 
SR21 | .09 .00 02 .05 -05 | -.05 | -05 | -.01 | -.01 -.02 -.06 -.01 
SR22 | .06 .04 04 .06 | -.03 | -.04 | -.06 .00 | -.02 01 -.02 .00 
SR23 | -.04 | -.05 .09 12 02 | -.03 | -.11 .03 01 02 -.01 .00 
SR24 | -.02 | -.07 .08 16 02 | -.03 | -.11 .02 01 .00 -.04 -.01 
SR25 | -.14 | -.07 13 14 .28 .03 -.06 09 .05 .06 .08 .05 
SR26 | -.17 | -.15 02 07 09 5 10 25 .08 .07 12 .05 
SR27 | -.07 | -.10 | -.07 | -.06 | -.01 11 23 18 .03 .07 10 .06 
SR28 | -.18 | -.12 01 04 09 13 08 30 .08 12 19 .07 
SR29 | -.04 | -.08 .04 .09 02 01 -.04 11 12 01 .04 .02 
SR30 | -.10 | -.07 04 .06 .05 04 -.01 13 .03 17 10 .05 
SR31 | -.11 | -.05 .05 .00 .06 .05 02 17 .03 09 17 .05 
SR32 | -.06 | -.07 .05 .08 04 02 | -.02 10 .04 05 .06 .04 


Note: Average r on-diagonal, Traditional = .23, Online = .15. 

Average r off-diagonal, Traditional = .04 Online = .02. 

Ns for Traditional and Online Courses = 2,215 to 2,301 and = 4,771 to 4,932, respectively. 
See Table 10 for item descriptions. 


Are the correlations between students’ ratings of progress on learning objectives and their 
ratings of the instructor’s teaching methods similar between types of courses? 


Table 19 presents correlations between students’ ratings of progress on the 12 learning 
objectives (“Obj 1” to “Obj 12”), the 20 teaching methods (“TM 1” to “TM 20”), and TM 47 
(instructor’s use of educational technology). The samples for these correlations included only 
instructors who rated a given objective as either important or essential. The pattern of 
correlations was very consistent across type of course. Furthermore, the teaching methods that 
were highly correlated with learning objectives (r > .60) closely followed the findings in Hoyt 
and Lee (2002a). One notable exception was found in the correlation between TM 47 (use of 
educational technology to promote learning) and Objective 7 (broad liberal education), which 
was slightly higher in online (r = .43) than traditional (r = .26) courses. Because of the high 
number of comparisons made among correlation coefficients, this difference was not considered 
meaningful. This demonstrates support for the use of IDEA as a diagnostic to guide 
improvement in online learning environments. 
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Table 19 
Correlations between Student Ratings on Learning Outcomes and Teaching Methods for 
Traditional and Online Courses 


Traditional 
Item Obj1 | Obj2 | Obj3 | Obj4 | Obj5 | Obj6 | Obj7 | Obj8 | Obj9 | Obj10 | Objil | Obj12 
™1 .68 67 72 .70 59 .66 .64 .60 62 65 .60 .68 
™ 2 .70 Fl .76 74 63 .70 .65 .64 67 .69 .64 73 
™ 3 .66 65 .69 .68 56 65 54 55 63 62 59 62 
™ 4 .74 75 77 75 58 67 .60 .60 63 .66 .64 .68 
™ 5 32 34 Al 38 .69 43 34 50 AY 51 44 A5 
™ 6 75 We) il AS 62 67 .63 62 65 .66 .64 .68 
™ 7 63 63 .69 .68 61 .70 65 62 61 63 59 .66 
™ 8 .70 72 74 73 63 .70 .62 .64 .70 .70 .69 1D 
™ 9 55 fap) 61 .60 35 55 40 61 74 .63 .63 .64 
™ 10 71 71 74 .70 56 .64 .60 63 64 65 65 .66 
™ 11 65 .64 67 .63 56 D2 46 55 25). .63 55 61 
™ 12 .70 .68 .69 .65 49 50 39 2 59 5 56 57 
™ 13 73 .74 77 74 61 71 .67 65 67 .70 .70 12 
™ 14 48 49 59 61 .68 .60 Al 53 .60 58 51 58 
T™ 15 67 .69 76 AS .70 76 64 67 74 72 .68 74 
™ 16 49 51 59 56 62 .60 53 67 62 73 .66 .66 
T™ 17 61 62 .64 .62 a3 57 50 ap) 58 58 55 59 
T™ 18 55 7 .63 .60 65 58 54 59 63 67 61 .66 
™ 19 337 59 65 61 58 .64 49 67 .68 65 .68 .68 
™ 20 65 65 .68 .67 61 63 58 65 65 61 61 69 
T™ 47 46 45 50 AT 38 36 26 39 54 44 43 48 
Online 
Item Obji | Obj2 | Obj3 | Obj4 | Obj5 | Obj6 | Obj7 | Objs | Obj9 | Obj10 | Objil | Objl2 
™ 1 63 .64 .68 .67 59 .63 54 61 62 64 65 67 
™ 2 64 65 69 .69 62 .63 58 62 65 .64 67 .69 
™ 3 63 62 65 .65 52 57 52 56 59 59 61 .60 
™ 4 .68 .68 ep 72 .60 .60 58 62 .64 65 67 .66 
™ 5 33 36 Al 39 63 46 37 A5 Al 44 47 A5 
™ 6 67 .68 71 71 62 .62 57 61 62 .64 67 65 
™ 7 57 58 .63 .62 .60 .60 56 63 59 61 62 62 
™ 8 .66 67 71 71 65 67 61 .68 67 .68 Aral .70 
™ 9 55 56 61 61 Pap) .60 50 61 .69 59 .64 62 
™ 10 .68 .68 71 71 7 61 59 62 63 .63 67 .64 
™ 11 59 .60 .68 .65 .60 53 43 55 56 .63 61 .60 
™ 12 65 .66 .68 .67 49 54 49 51 57 55 59 56 
™ 13 67 .69 12 71 62 65 .62 65 .64 .68 71 67 
™ 14 AT 49 58 58 .64 58 43 55 moi 56 58 55 
T™ 15 .64 .66 72 2: 67 .69 59 .69 .69 71 71 .70 
™ 16 46 49 56 55 62 Rey 52 59 55 61 62 .60 
T™ 17 57 58 61 .60 50 56 53 aD Bae) 54 56 56 
T™ 18 48 50 56 56 .66 56 48 56 54 59 59 .60 
™ 19 58 .60 .68 .66 61 67 59 67 65 65 fl .64 
™ 20 58 .60 .64 .64 61 .60 50 59 .60 61 .60 63 
TM47 57 56 58 57 AT 48 43 49 56 52 53 53 


Note: Ns for Traditional and Online Courses = 3,706 and = 10,833, respectively. 
See Table 10 for item descriptions. 
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Are the correlations between students’ characteristics (e.g., work habits, motivation), 
overall ratings of the course and the instructor, and perceived progress on relevant 
objectives similar across types of courses? 


In the IDEA Diagnostic Form Report, students’ ratings of the instructor, the course, and 
their progress on relevant objectives (PRO) are adjusted for their correlations with student/course 
characteristics. Therefore, it is important to investigate the similarity of those correlations across 
type of course. Table 20 presents correlations among these variables. The pattern of correlations 
was very similar across course modalities with a few exceptions. First, students’ adjusted self- 
reported progress on relevant objectives (Adjusted TSCORE PRO) was somewhat more highly 
correlated with the instructor’s use of technology (D47) in online (r = .50) than traditional (r = 
.35) courses. In both cases, however, the correlation was moderately positive. Second, students’ 
ratings of the excellence of the course was somewhat more highly correlated with their course 
effort (D37) in traditional (r = .42) than online (r = .27) courses. However, regardless of course 
type, the relationship was low to moderate and positive. Third, the correlations between 
difficulty of the subject matter (D35) and two global measures (D40/S16, D42/S18) were weak 
and positive in traditional courses. In contrast, those correlations were weak and negative in 
online courses. However, in both cases the relationships were almost negligible. No other 
correlations were meaningfully different across types of courses. In general, then, there were 
more similarities than differences in the magnitude and direction of correlations across online 
and traditional courses. 


23 


Table 20 


Inter-Correlations between Student/Course Characteristics and Summary Judgment Items 


Student Item 1 2 3 4 5 6 7 8 9 10 11 12 13 
Traditional Courses 
1. Instructor use of technology (D47) 1 
2. Amount of reading in class (D33) 0.20 1 
3. Amount of other work (D34) 0.23 0.30 1 
4. Difficulty of subject (D35) 0.07 0.39 0.58 1 
5. Strong desire to take course (D36) 0.27 0.05 0.16 0.14 1 
6. Work harder on course (D37) 0.27 0.32 0.64 0.63 0.44 1 
7. Wanted to take course from instructor (D38) 0.35 0.10 0.18 O19 0.55 0.42 1 
8. Wanted to take course regardless (D39/S15) 0.22 0.04 0.18 O.11 0.71 0.37 0.28 1 
9. Put forth more effort in all classes (D43/S13) 0.24 O15 0.28 0.22 0.29 042 0.29 0.29 1 
10. Positive feelings toward field (D40/S16) 0.45 0.10 0.18 O11 O71 044 0.68 0.52 0.30 1 
11. Excellent Teacher (D41/S17) 0.51 0.10 012 0.08 044 0.32 0.72 0.29 0.21 0.76 1 
12. Excellent Course (D42/S18) 0.46 O11 O16 O10 063 042 0.67 046 0.27 0.85 0.83 1 
13. Raw TSCORE PRO 0.49 0.19 0.22 0.16 049 044 066 035 0.35 0.78 0.79 0.80 1 
14. Adjusted TSCORE PRO 0.35 0.13 0.08 0.16 0.24 0.29 0.54 -0.01 -0.06 0.59 0.70 0.64 0.86 
Online Courses 
1. Instructor use of technology (D47) 1 
2. Amount of reading in class (D33) 0.11 1 
3. Amount of other work (D34) 0.20 0.45 1 
4. Difficulty of subject (D35) 0.08 0.45 0.52 1 
5. Strong desire to take course (D36) 0.30 0.03 0.02  -0.04 1 
6. Work harder on course (D37) 0.28 0.36 0.54 0.53 0.32 1 
7. Wanted to take course from instructor (D38) 0.44 0.05 0.10 0.09 0.49 0.33 1 
8. Wanted to take course regardless (D39/S15) 0.18 0.03 0.04 -0.02 0.66 0.24 0.17 1 
9. Put forth more effort in all classes (D43/S13) 0.25 0.14 0.23 0.16 0.24 0.33 0.25 0.18 1 
10. Positive feelings toward field (D40/S16) 0.51 0.05 0.04 -0.05 0.66 0.31 0.60 0.47 0.24 1 
11. Excellent Teacher (D41/S17) 0.65 0.03 0.06 0.01 040 0.26 0.65 0.26 0.17 0.71 1 
12. Excellent Course (D42/S18) 0.58 0.03 0.03 -0.05 0.56 0.27 0.61 039 0.21 0.81 0.85 1 
13. Raw TSCORE PRO 0.60 0.09 0.13 0.02 045 033 0.58 0.29 0.25 0.74 0.77 0.78 1 
14. Adjusted TSCORE PRO 0.50 0.04 0.03 0.06 0.21 0.24 048 -0.05 -0.13 0.55 0.67 0.64 0.87 


Note: Ns for Traditional and Online Courses = 3,704 to 5,272 and = 10,833 to 13,365, respectively. 
The letter and number in parentheses indicates the number of item on the Diagnostic (D) and Short (S) Forms respectively. 
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Are the correlations between student ratings of teaching methods and overall measures of 
effectiveness similar in online and traditional courses? 


Table 21 presents correlations between student ratings of how frequently the instructor 
used each of 20 teaching methods, three global ratings of teaching effectiveness, and progress 
on relevant objectives (Raw PRO, Adj PRO). The three global ratings of teaching effectiveness 
were: “As a result of taking this course, I have more positive feelings toward this field of study” 
(D40/S16); “Overall, I rate this instructor an excellent teacher” (D41/S17); “Overall, I rate this 
course as excellent” (D42/S18). The pattern of correlations in Table 21 is very consistent across 
traditional and online courses. 


Table 21 
Inter-Correlations between Student Ratings of Teaching Methods and Summary Judgment Items 
or Traditional and Online Courses 


Traditional Courses Online Courses 

Tein D40/ | D41/ | D42/ | Raw | Adj D40 D41 D42 Raw Adj 
S16 S17 S18 | PRO | PRO /S16_ | /S17 | /S18 | PRO | PRO 

T™ 1 .67 | .83 69 I 3dS 62 62 85 “hd i 62 
TM 2 70 | .85 13 17 | .66 62 .83 eg 73 .64 
T™ 3 62 | .77 | .69 | .70 | .60 my ato .68 .68 9 
T 4 YS ao .76 | .77 | .66 67 82 sD 74 .63 
T 5 35 | 41 34 | 42 | 31 36 43 39 42 33 
T™ 6 th 2° ||, 20D ee) BOF .66 719 74 .73 62 
TM 7 64 | .76 | 66 | .71 .60 54 ihe. .63 65 JD 
T™ 8 68 | .79 | .71 TE \. 367 .62 78 270 we .63 
Ti 9 54 | .62 | 56 | .63 50 54 .67 .60 .64 wo 
T™ 10 .70 87 .76 75 .66 .66 85 77 74 65 
TM 11 64 | .68 | .61 66 | .55 .63 .69 .66 .66 ae) 
TM 12 PS fc | Me | 64 | 67 | .57 .60 74 .70 .69 .60 
T™ 13 ES | SAP | 768 | 378) | OT .69 719 74 74 62 
T™ 14 52 54 51 7 41 52 56 55 58 44 
TM 15 AEOe ||) hon. E 78 | .63 .64 ad yal Afi) .63 
TM 16 Ot ll £08 56 | 61 48 4 59 pays, 58 46 
T™ 17 soe |) As |). 3 66 | .57 52 eT 65 .64 58 
TM 18 58 | .67 | 58 | 66 | .54 A8 61 5 58 50 
TM 19 60 | .67 | 62 | 66 | .52 09 69 .66 .69 ih 
TM 20 63 — |, 575 65 mel .60 56 .76 65 67 58 
TM 47 AS< | 251 AG. | 349° 4 235 1 65 58 .60 50 


Note: Correlations between traditional and online courses that were |> .15| are bolded. 

Ns for Traditional and Online Courses = 3,706 and = 10,833, respectively. 

The letter and number in parentheses indicates the number of item on the Diagnostic (D) and Short (S) 
Forms respectively. See Table 20 for item detail. 

RAW PRO = Raw Score PRO, Adj PRO = Adjusted Score PRO. 

See Table 10 for item descriptions. 


The 20 teaching methods on the Form are combined to form five scales for describing 
teaching approaches: Stimulating Student Interest, Fostering Student Collaboration, 
Establishing Rapport, Encouraging Student Involvement, and Structuring Classroom 
Experiences. In IDEA Research Note #1 (The IDEA Center, 2003), the five teaching 
approaches served as explanatory variables in regression analyses performed individually on 
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the three global ratings. Students’ ratings on the five scales were more highly correlated with 
ratings on the “excellent teacher” item (R” = .85) than on “increased positive feeling” (R° = .64) 
and “excellent course” (R° = .73). For the current report, we conducted the analyses reported in 
Research Note #1 separately for traditional and online courses. For traditional courses, the five 
scales explained more variance in the “excellent teacher” item (R* = .83) than in the “increased 
positive feeling” (R° = .60) and “excellent course” (R° = .66) items. For online courses, the 
pattern was the same: “excellent teacher” (R° = .82), “increased positive feeling” (R° = .52), and 
“excellent course” (R° = .67). So, in both traditional and online courses, the extent to which 
students regard the instructor as “excellent” has much to do with teaching approaches. 


In Research Report #4, Hoyt and Lee (2002b) reported the results of multiple regression 
analyses conducted on the 12 learning objectives with the five teaching approaches as 
explanatory variables. From those analyses, the authors proposed six teaching styles (A through 
F), each of which places different emphasis on the five teaching approaches, depending upon 
the specific learning objective. We investigated whether these teaching styles would be similar 
across different course modalities. The samples for these regression analyses included only 
instructors who rated a given objective as either important or essential. 


Teaching Style A. Appropriate for helping students to pursue cognitive learning 
objectives (Obj. 1 and 2) and to make applications of learning (Obj. 3 and 4), Teaching Style A 
emphasizes stimulating student interest. The standardized beta coefficients presented in Table 
22 confirm the consistency in this style across traditional and online courses, as the coefficients 
are high for stimulating student interest. In addition, structuring the classroom experience is 
associated with progress on these objectives. 


Table 22 

Standardized Beta Coefficients for Explanatory Variables in Teaching Style A Objectives 
Objectives for Stimulating Foster Establish Encourage Structure 
Teaching Style A Student Interest Collaboration Rapport Involvement Classroom 


Trad. Online Trad. Online Trad. Online Trad. Online Trad. Online 
pene 0.45 040 -0.19 -0.14 0.02 -0.02 0.10 0.03 0.40 0.45 
knowledge 
2. Learn principles, 
theories 
3. Apply course 
material 
4. Professional skills, 
attitudes 


0.54 0.45 -0.14 -0.10 -0.06 -0.02 0.05 0.02 037 041 
0.41 0.32 -0.09 -0.09 0.07 O01 017 O22 0.29 0.35 


0.42 0.42 -0.16 -009 O15 002 O18 O12 022 0.32 


Teaching Style B. Table 23 shows the results of multiple regression analyses performed 
on Teaching Style B objectives, which emphasize in-depth analysis and thought (Obj. 11) as 
well as values development (Obj. 10). As in Teaching Style A, this style highlights stimulating 
student interest in both modalities and, to some extent, fostering student collaboration in 
traditional courses. Encouraging involvement (e.g., encouraging students to use multiple 
resources, involving students in “hands-on” activities) is moderately helpful in both types of 
courses when attempting to foster critical analysis and evaluation. 
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Table 23 
Standardized Beta Coefficients for Explanatory Variables in Teaching Style B Objectives 


Objectives for Stimulating Foster Establish Encourage Structure 
Teaching Style B Student Interest _ Collaboration Rapport Involvement Classroom 


Trad. Online Trad. Online Trad. Online Trad. Online Trad. Online 


Ie values 0.45 0.52 0.26 0.04 -0.02 0.10 0.08 0.11 0.07 -0.02 
development 
Bhs Cuncal analysis. -Jg.46 044 0.18 0.06 -0.15 -0.01 014 019 016 0.11 
evaluation 


Teaching Style C. This teaching style stresses helping students to achieve “general 
education” objectives, such as gaining a broad liberal education (Obj. 7) and increasing interest 
in learning (Obj. 12). The style is similar to Style B, except that establishing rapport becomes 
somewhat more important. The standardized beta coefficients in Table 24 indicate this is the case 
for both traditional and online courses with respect to increasing interest in learning. Moreover, 
in online courses, structuring the classroom experience is associated with greater student 
progress in achieving a broad liberal education. 


Table 24 

Standardized Beta Coefficients for Explanatory Variables in Teaching Style C Objectives 
Objectives for Stimulating Foster Establish Encourage Structure 
Teaching StyleC Student Interest Collaboration Rapport Involvement Classroom 


Trad. Online Trad. Online Trad. Online Trad. Online Trad. Online 


7, Broad liberal 0.49 0.56 0.07 0.06 0.40 0.01 -0.20 -0.16 -0.06 0.20 


education 


12. Increased 0.38 0.39 0.05 0.05 0.23 0.23 018 0.09 -0.01 0.03 
interest in learning 


Teaching Style D. Progress on objectives related to self-expression - developing creative 
capacities (Obj. 6) and gaining communication skills (Obj. 8) — is associated with Style D 
teaching. Stimulating student interest is most helpful, followed by establishing rapport and 
encouraging involvement. Table 25 indicates this was true for both traditional and online 
courses. 


Table 25 

Standardized Beta Coefficients for Explanatory Variables in Teaching Style D Objectives 
Objectives for Stimulating Foster Establish Encourage Structure 
Teaching Style D Student Interest Collaboration Rapport Involvement Classroom 


Trad. Online Trad. Online Trad. Online Trad. Online Trad. Online 
6. Creative capacities 0.53 0.31 -0.01 0.04 0.23 0.18 0.10 0.20 -0.06 0.02 


eo 0.16 0.39 0.19 0.06 017 015 O21 017 0.06 -0.02 


Teaching Style E. This teaching style is especially helpful when students need to acquire 
team skills (Obj. 5). Instructors foster collaboration, which is supported by stimulating student 
interest and encouraging involvement. This pattern is consistent across course modalities, as 
indicated in Table 26. Establishing rapport is also moderately important for building team skills 
when teaching an online course. 
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Table 26 
Standardized Beta Coefficients for Explanatory Variables in Teaching Style E Objectives 


Objectives for Stimulating Foster Establish Encourage Structure 

Teaching StyleE Student Interest Collaboration Rapport Involvement Classroom 
Trad. Online Trad. Online Trad. Online Trad. Online Trad. Online 

5. Team skills 0.17 0.22 0.52 039 0.03 018 0.09 0.16 -0.01  -0.17 


Teaching Style F. This teaching style is most closely associated with courses that stress 
helping students to learn how to find and use resources (Obj. 9). Encouraging involvement is 
most essential, along with placing some emphasis on stimulating student interest. The results of 
the regression analyses were, for the most part, consistent across traditional and online courses, 
as indicated in Table 27. The only noticeable difference is that establishing rapport may be 
somewhat more important when teaching an online course. 


Table 27 

Standardized Beta Coefficients for Explanatory Variables in Teaching Style F Objectives 
Objectives for Stimulating Foster Establish Encourage Structure 
Teaching StyleF Student Interest Collaboration Rapport Involvement Classroom 


Trad. Online Trad. Online Trad. Online Trad. Online Trad. Online 
9. Finding and 0.26 0.26 0.11 -0.07 -0.07 0.13 035 033 0.18 0.10 


using resources 


CONCLUSIONS 


The results of this study reveal more similarities than meaningful differences between 
IDEA student ratings in traditional and online courses. First, instructors’ ratings of the 
importance of the 12 IDEA learning objectives did not vary much between the types of course. 
Average instructor ratings and the percent rating each objective as essential or important were 
very similar. Second, the pattern of inter-correlations among the learning objectives is 
remarkably similar across course modalities. Third, student’s progress on relevant objectives and 
global ratings of instructor/course effectiveness are similar in online and traditional courses. 
Students identify good teaching when they see it, whether it occurs online or face to face. Fourth, 
students consistently report greater progress on objectives the instructor rates as important or 
essential no matter which instructional format is used. Moreover, the highest correlations 
between instructor ratings of importance and students’ ratings of progress are found in their 
ratings of the same objectives. This holds true after removing Short Form users, novice users, 
and classes with fewer than 10 students responding. Student ratings of progress, therefore, are 
valid in both course settings. Fifth, students’ ratings of how frequently the instructor used 20 
teaching methods do not vary meaningfully between course formats. Online and on-campus 
instructors are perceived to use the methods with similar frequency. Sixth, the pattern of 
correlations between students’ ratings of progress on the learning objectives and their ratings of 
the instructor’s use of 20 teaching methods are highly similar between both types of courses. 
Suggestions for improving teaching effectiveness, based on these relationships, are supported in 
both online and face-to-face formats. Furthermore, those correlations are, for the most part, 
comparable to those reported in Hoyt and Lee (2002a). Eighth, the correlations between 
student/course characteristics and global measures of effectiveness are very similar for students 
in online and traditional courses with only a few minor exceptions. Evidence for the validity of 
adjustments to raw scores, based on these relationships, can therefore be found in both course 
formats. Finally, the six teaching styles reported in Hoyt and Lee (2002b) are quite comparable 
across traditional and online courses. 
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However, there are some minor differences worth noting. First, moderate differences are 
found in response rate, as students in traditional courses are somewhat more likely to complete 
ratings. Second, students in online courses report their instructors use educational technology 
more frequently than do those in traditional courses. Third, the correlation between instructor use 
of educational technology and students’ self-reported progress on relevant objectives is slightly 
higher in online courses than traditional courses. So, as one might expect, using educational 
technology with greater frequency is slightly more important in online courses. Fourth, students 
in online courses report somewhat more reading (Item 33) and somewhat less motivation to take 
the course from the instructor, although these differences are small. Fifth, fostering student 
collaboration is more helpful in traditional courses when the focus is on values development and 
critical analysis/evaluation. Sixth, structuring the classroom environment may be somewhat more 
important in online courses if the instructor wishes to help students achieve a broad liberal 
education. Finally, when helping students to find and use resources, establishing rapport may be 
somewhat more important in online courses. 


Overall, then, the current findings indicate the IDEA Student Ratings System is 
appropriate for both online and traditional courses. The minor differences observed between 
online and traditional courses ultimately may help guide instructors for improving student 
learning experiences in both teaching environments. 


However, The IDEA Center recognizes that no single survey can anticipate the unique 
needs of every learning environment. The use of additional questions may be helpful in 
addressing areas not covered in the IDEA instrument, but important to a particular course or 
learning environment. Appendix B contains the handout, Using Additional Questions for Online 
Learning Environments, which can serve as a guide to maximizing the feedback obtained 
through the IDEA Student Ratings of Instruction System. 
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=m nm | nm = 
a i a A a | 
- | Faculty Information Form IMPORTANT! leceece| 

= 

_ @ IDEA, See Directions to Faculty: west carpe 


Institution: Instructor: 
Time and Days Class Meets: 


Objectives: Using the scale provided, identify the relevance of each of the twelve objectives to this 

course. As a general rule, prioritize what you want students to learn by selecting no more than 3-5 

objectives as either Important or Essential. The weighting system used to generate the IDEA report 

weighs Essential objectives *2," Important objectives *1," and Minor objectives *0." 

(Scale - M = Minor or No Importance, | = Important, E = Essential) 

MIE 

1. OOO Gaining factual knowledge (terminology, classifications, methods, trends) 

2. OOO Learning fundamental principles, generalizations, or theories 

3. OOO Leaming to apply course material (to improve thinking, problem solving, and decisions) 

4. OOO Developing specific skills, competencies, and points of view needed by professionals in 
the field most closely related to this course 

5. OOO Acquiring skills in working with others as a member of a team 

6. OOO Developing creative capacities (writing, inventing, designing, performing in art, music, 
drama, etc.) 

7. OOO Gaining a broader understanding and appreciation of intellectual/cultural activity (music, 
science, literature, etc.) 

8. OOO Developing skill in expressing oneself orally or in writing 

9. OOO Leaming how to find and use resources for answering questions or solving problems 

10. OOO Developing a clearer understanding of, and commitment to, personal values 
11. OOO Leaming to analyze and critically evaluate ideas, arguments, and points of view 
OO) g q 


@™ Contextual Questions (Research Purposes): 
=== s The IDEA Center will conduct research on these optional questions in order to improve the interpretation of student ratings 


. - approaches 5 this of its requirements respect 
represents rhe rats I é the features listed below. Use the following code to alin your 

approach to this course? res. ses: 

(Mark only one) secondary approach’ Hy . Sone (or we required 

.- e required 
{Mark:only one) M = Much required 

= Lecture NSM 
= Discussion/recitation OOOA. Writing 
= Seminar Oral communication 
= Skill/activity Computer applications 
= Laboratory . Group work 
= Field Experience . Mathematical/quantitative work 
= Studio OF. Critical thinking 
= Multi-Media OOO G Creative/artistic/design endeavor 


= Practicum/clinic OOOH. Reading 


= Lecture 

= Discussion/recitation 
= Seminar 

= Skill/activity 


= Field Experience 
= Studio 
= Multi-Media 
(0) = Practicum/clinic 
(0) = Other = Other 
GM rr5901 (11/08) 


© 
© 
© 
@ 
© = Laboratory 
© 
0) 
© 


©QOOOOOOOO 


z 
= 
¥ 
$ 
5 
3 
= 


Continue on back page 
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Contextual Questions Continued: 


Rate each of the circumstances listed below, using the following 5. 
code to respond: 


Please identify the principal type of student 
enrolling in this course 


P = Had a positive impact on learning 

| = Neither a positive nor a negative impact 

N = Had a negative impact on learning 

? = Can't judge 
rPEeY? 
O0004. 
O0008 
O000¢ 


O000® 


First-year students/sophomores seeking to 
meet a “general education" or “distribution* 
requirement 

First-year students/sophomores seeking to 
develop background needed for their 
intended specialization 

Upper level non-majors taking the course 
as a “general education" or “distribution* 
requirement 

Upper level majors (in this or a related 
field of study) seeking competence or 
expertise in their academic/professional 
specialty 

Graduate or professional school students 
Combination of two or more of the above 
types 


Physical facilities and/or equipment 

Your previous experience in teaching this course 
Substantial changes in teaching approach, course 
assignments, content, étc. 

Your desire to teach this course 

Your control over course management decisions 
(objectives, texts, exams, etc.) 

Adequacy of students’ background and preparation for 
the course 

Student enthusiasm for the course 

Student effort to lear 

Technical/instructional support 


6. Is this class: 


a. Team taught? 
b. Taught through distance learning? 


O Yes 
O Yes 


0100 
0200 


Discipline Codes (Modified CIP Codes) 


Agricultural Business and Production 
Agricultural Sciences 


9902 


Developmental Reading 
Developmental Writing 


2700 


Mathematics and Statistics 


Music (Performing, Composing, 
Theory) 


0300 Conservation and Renewable Natural 9904 Developmental Natural Sciences 
Resources ; 5116 Nursing 
4506 Economics 
0400 Agricultural and Related Programs : 3100 Parks, Recreation, Leisure, and 
1300 Education Fitness Studies 
0500 Area Ethnic and Cultural Studies 
1400 Engineering 3801 Philosophy 
5007 Art (Painting, Drawing, Sculpture) - 
ae 1500 Engineering-Related Technologies 4000 Physical Science (EXCEPT 
sic is 
9910 English as Second Language Piysiceand Chemistry) 
2600 Biological Sciences/Life Sciences 
es 2301 English Language and Literature 4008 Physics 
5201 Business, General 
5000 Fine and Applied Arts (EXCEPT 4510 Political Science and Govemment 
5202 Business Administration and Management Art, Music, and Design and Applied = 4200 Psychology 
Arts) 
5203 Business - Accounting alata i — 4400 Public Administration and Services 
‘oreign ages a eratures CEPT 
5208 Business - Finance i large Ee SORENSON 
3105 Health and sical i 
5212 Business Information and Data gant: Ege a a 
sapiansiatginonaiind 5100 Health Professions and Related Te tena. ane 
by Economics, History, Political 
5214 Business - Marketing Sciences (EXCEPT Nursing) Science, and Sociology) 
4005 Chemistry 5199 Health Professions and Related 4407 Social Work and Service 
0900 Communications canoe ci bie onl 4511 Sociology 
1100 Computer and Information Sciences 4508 History 2310 Speech and Rhetorical Studies 
1900 Human Sciences/Family and 
4301 Criminal Justice and Corrections Consumer Sciences Vocational/Technical Programs 
see Website tment codes 
1205 Culinary Arts and Related Services 2400 Liberal Arts & Sciences, General Seso- 40008 Cone 
1103 Processing Tech " Studies and Humanities 
po hs s ne 9900 Other (to be used when none of the 
2200 General Legal Studies above codes apply) 
5004 Design and Applied Arts (Undergraduate) 
9901 Developmental Math 2500 Library Science 


To see an expanded list of discipline codes go to: www.theideacenter.org/DisciplineCodes 
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SURVEY FORM - STUDENT REACTIONS TO INSTRUCTION AND [area 
COE eontanr «<nsnsnam pase 


Institution: Instructor: 


Course Number: Time and Days Class Meets: 


Your thoughtful answers to these questions will provide helpful information to your instructor. 


Describe the frequency of your instructor's teaching procedures, using the following code: 
1=Hardly Ever 2=Occasionally 3=Sometimes 4=Frequently 5=Almost Always 


= 
=> 
o 
> 
o 
= 
2 
c 
2 
= 
° 
= 


= 14 @© © @®_ © Displayed a personal interest in students and their learning 
m= 2.) © @® @©®_ © Found ways to help students answer their own questions 
m= 304) © © @®_ © Scheduled course work (class activities, tests, projects) in ways which encouraged students to stay up-to-date in their work 
m= 404) @ © @®_ © Demonstrated the importance and significance of the subject matter 
m5.) @ © © _ © Formed “teams" or “discussion groups" to facilitate learning 
m 6.0) @ © @®_ © Made it clear how each topic fit into the course 
m= 7.4) © @® @®_ © Explained the reasons for criticisms of students’ academic performance 
m= 8.0) @© © @©®_ © Stimulated students to intellectual effort beyond that required by most courses 
m9.0) @ @® ©®_ © Encouraged students to use multiple resources (e.g. data banks, library holdings, outside experts) to improve understanding 
m10.0 © © @®_ © Explained course material clearly and concisely 
=1144 @ @© @®_ © Related course material to real life situations 
m120 © © ®_ © Gave tests, projects, etc. that covered the most important points of the course 
m=13.0) © © @®__ & Introduced stimulating ideas about the subject 
m14.4) @ © @®_ @ Involved students in "hands on* projects such as research, case studies, or “real life" activities 
m15.4) @ © © __ © Inspired students to set and achieve goals which really challenged them 
m16.0) @ @® @©®_ © Asked students to share ideas and experiences with others whose backgrounds and viewpoints differ from their own 
=17.4) @ @® @®_ ©@® Provided timely and frequent feedback on tests, reports, projects, etc. to help students improve 
m18.0) @ @© © _ © Asked students to help each other understand ideas or concepts 
m19.0 © @©® @®_ © Gave proects, tests, or assignments that required original or creative thinking 
m20.0) @ @® @®_ © Encouraged student-faculty interaction outside of class (office visits, phone calls, e-mail, etc.) 
= 
= 
— Twelve possible learning objectives are listed below, not all of which will be relevant in this class. Describe the 
ani amount of progress you made on each (even those not pursued in this class) by using the following scale: 
= 
= 1-No apparent progress , - 
2-Slight progress; | made small gains on this objective. 
= 3-Moderate progress; | made some gains on this objective. 
= 4-Substantial progress; | made large gains on this objective. 
_ 5-Exceptional progress; | made outstanding gains on this objective. 
= 
= 


mm Progress on: 


m21.0) © © ©®_ © Gaining factual knowledge (terminology, classifications, methods, trends) 

m22.2 © © @®_ © Leaming fundamental principles, generalizations, or theories 

m23.0) @ © @_ © Leaming to apply course material (to improve thinking, problem solving, and decisions) 

m240) © © ©®_ ©& Developing specific skills, competencies, and points of view needed by professionals in the field most closely 
= related to this course 

m25.0) © @ @_ © Acquiring skills in working with others as a member of a team 

m26.4) @© © @®_ © Developing creative capacities (writing, inventing, designing, performing in art, music, drama, etc.) 

=m27. @ © @©®_~ © Gaining a broader understanding and appreciation of intellectual/cultural activity (music, science, literature, etc.) 
m=28. © © @__ © Developing skill in expressing myself orally or in writing 

m29.0 @ © @_ © Leaming how to find and use resources for answering questions or solving problems 

m30.4) © © @®_ © Developing a clearer understanding of, and commitment to, personal values 

m31.4) © © @_ © Leaming to analyze and critically evaluate ideas, arguments, and points of view 

32.9 @ © @_ © Acquiring an interest in leaming more by asking my own questions and seeking answers 

= 

= 

™@™ Copyright © IDEA Center, 1998 Continued on back page 
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On the next three items, compare this course with others you have taken at this institution, using the following code: 


1=Much Less than 2=Less than 3=About Average 4=More than 5=Much More 
Most Courses Most Courses Most Courses than Most Courses 


The Course: 


3.0 @ @® © _ © Amountof reading 
34.0 © @® @®_~ © Amount of work in other (non-reading) assignments 


5.0 @©@ © @®_ © Difficulty of subject matter 


Describe your attitudes and behavior in this course, using the following code: 


1=Definitely 2=More False 3=In Between 4=More True 5=Definitely 
False Than True Than False True 


3.0 @ © @®_ © thada strong desire to take this course. 

37.Q @ @® @®@_ © | worked harder on this course than on most courses | have taken. 

3.0 @ @® ©®_ © [really wanted to take a course from this instructor. 

39.4 @ © @® _ © [really wanted to take this course regardless of who taught it. 

40.0 @ @® @® © Asa result of taking this course, | have more positive feelings toward this field of study. 
4.0 @© @® @®_~ © Overall, | rate this instructor an excellent teacher. 

42. @® @®© @®~ © Overall, | rate this course as excellent. 


For the following items, blacken the space which best corresponds to your judgment: 


1=Definitely 2=More False 3=In Between 4=More True 5=Definitely 
False Than True Than False True 


BQ © © @® © Asanie, | put forth more effort than other students on academic work. 

4.Q @ @® @®_ © Mhe instructor used a vanety of methods--not only tests--to evaluate student progress on course objectives. 
4.0 © @© @®_ © The instructor expected students to take their share of responsibility for leaming. 

4.4) @ @® _ @®_ © Mhe instructor had high achievement standards in this class. 

47.2 @ @®_ ®_ © The instructor used educational technology (e.g., Internet, e-mail, computer exercises, multi-media 


presentations, etc.) to promote leaming. 


EXTRA QUESTIONS 
If your instructor has extra questions, answer them in the space designated below (questions 48-67): 


rs 3 a 6 . Ye 3 3 . 3 Use the space below for comments 
009 © © @ © 00 © O@ © (unless otherwise directed). 

10 @ © © © 109 ©2@6®@ ® Note: Your written comments may be 
52.0 © © © © 20 © © © @ retumed to the instructor, You may want 
30 @ © @ © 820 © @ @ © to PRINT to protect your anonymity. 
40 ©@ ©@ © © 4Q © © @ © 

5O © @ ®@ © 56O ©@ @ ®@ © 

50 © © © © 60 © @ @ ®@ 

70 © © © © 70 © © @ © 


Comments: 


T5903 (08/08) 67654 Printed in USA Di 
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SHORT FORM - STUDENT REACTIONS TO INSTRUCTION AND COURSES 


Institution: Instructor: 
Course Number: Time and Days Class Meets: 


Improper M. 
OVBOS 


tHE N02 PRINCI ONLY, Ian 


amount of progress you made on each (even those not pursued in this class) by using the following scale: 


1-No apparent progress 

2-Slight progress; | made small gains on this objective. 
3-Moderate progress; | made some gains on this objective. 
4-Substantial progress; | made large gains on this objective. 
5-Exceptional progress; | made outstanding gains on this objective. 


OITA 


Twelve possible learning objectives are listed below, not all of which will be relevant in this class, Describe the 


) 
literature, etc.) 


Progress on: 
10 @© © © > © Gaining factual knowledge (terminology, classifications, methods, trends) 
20 @ @® @®_ © Leaming fundamental principles, generalizations, or theories 
30 © @® @®_ © Leamingto apply course material (to improve thinking, problem solving, and decisions) 
4.@ ® ® ©® © Developing specific skills, competencies, and points of view needed by professionals in the field most closely 
related to this course 
5.Q@ @ @® @_ © Acquiring skills in working with others as a member of a team 
6£© @ @® @®_ © Developing creative capacities (writing, inventing, designing, performing in art, music, drama, etc 
70 @© @®© @®_ © Gaining a broader understanding and appreciation of intellectual/cultural activity (music, science, 
8&Q © @® @®_ © Developing skill in expressing myself orally or in writing 
1Q @©@ @© ® _ © Leaming how to find and use resources for answering questions or solving problems 
m10. @ @® @®_~ © Developing a clearer understanding of, and commitment to, personal values 
m1. © © @®_~ © Leamingto analyze and critically evaluate ideas, arguments, and points of view 
=12.0 @ © @®_~ ©& Acauiring an interest in learning more by asking my own questions and seeking answers 


For the remaining questions, use the following code: 


1=Definitely 2=More False 3=In Between 4=More True 5=Definitely 
False Than True Than False True 


213.0 © © © © Asanie, | put forth more effort than other students on academic work. 

m140) @ © @®_ ® Mybackground prepared me well for this course's requirements. 

mm 15.0) © @®_ © [really wanted to take this course regardless of who taught it. 

=16 @ @©® @®_ ©® Asaresult of taking this course, | have more positive feelings toward this field of study 
@® @®© @®~ © Overall, | rate this instructor an excellent teacher. 
® ® ®@ 


©) Overall, | rate this course as excellent 


= 
- EXTRA QUESTIONS 
-_ If your instructor has extra questions, answer them in the space designated below (questions 19-38), 


219.0 © © © 
m2. © © @ 
m21.0 © © @ 
m2. © © @ 
=230 © © @ 


70 © © @ & 20 © © @ & 37.© 


©QOOOO® 


@™ Copyright © IDEA Center, 2002 Use the space provided on the back of this form for your comments. 


4QO ©@OOGO BO OOOO 4O OD O @ 
a2O © @®@ © © MO © OO OH 850 © © @ 
26. ®OO ADOT OOH #O © OD @ 


20 0006 820 O0OOO BO OOD 


©QO©OO® 
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Appendix B 


Using Additional Questions for Online Courses 
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Using Additional Questions for 


Online Courses 


Question Areas 


Course Design/ 
Course Materials 


Online Activities 


Interactions with 
Instructor 


Student Interactions 


Student 
Characteristics 


environments. The IDEA system, which 
asks students to (a) rate their progress on 12 
different course objectives, and (b) rate the 
frequency with which their instructor 
employs each of 20 teaching "methods," 
offers the instructor the option of asking up 
to 20 additional questions on either the 
Diagnostic Form or Short Form. The 
instructor may wish to ask questions that 
pertain to the special characteristics of his/ 
her course which were not asked by any of 
the standard items. The following provides 
suggestions for areas that might be 
important to online learning environments 
but not addressed in the standard IDEA 
instrument. There is no one correct way to 
address these areas, so in many cases, 
multiple options are provided for your use 
or adaptation. The class report will provide 
the frequencies of student responses and the 
average (mean) for each additional question. 


| No single survey form can anticipate the Please keep a record of the questions you 
needs of all instructors or learning included. 


Unless specifically noted, you might use 
one of the following sets of response 
options for the items in these lists . 


OPTION A 


1= Hardly Ever 
2= Occasionally 

3 = Sometimes 

4 = Frequentl 

5 = Almost A ways 


OPTION B 

1= Strongly Disagree 

2 = Disagree 

3 = Neither Agree nor Disagree 
4 = Agree 

5 = Strongly Agree 


OPTION C 


1 = Definitely False 

2 = More False than True 
3 = In Between 

4 = More True then False 


5= Definitely True 


| Managing Additional Questions in IDEA Online 


Instructor Use of 


3 
Technology Instructors can only add additional questions to a course before the start date of the student 
y q 
surveys. When the Faculty Information Form is emailed to the instructor, a link to 
Technology and 3 “Manage Additional Questions” is provided at the bottom of the welcome screen. Twenty 
Learning 


total additional questions can be added (e.g., if 5 institutional questions are used, an 


instructor can add 15). The tutorial for Adding Additional Questions in IDEA Online 


illustrates the steps for faculty to add their own courses into IDEA Online. 


| Course Design/ Course Materials 


The expectations for this course were clearly outlined at the beginning of the course. *** 


Technology Support 3 
Overall Satisfaction 4 


Learning Outcomes 4 


Open-ended/Free Vi The course materials are easily accessible. * 
Response 


I was able to understand and follow the course navigation structure. * 
The instructions for accessing resource materials were understandable. * 
Overall, the course materials were easy to use. 


Tutorial: 
The [insert course component] was easy to use. 


Adding Questions 
in IDEA Online 


The [insert course component] supported my learning. 


USING ADDITIONAL QUESTIONS FOR ONLINE COURSES 


RESPONSES 


1 = None 

2= A Little (1-25%) 

3 = Some (25-50%) 

4 = Very Much (50-95%) 
5 = All (95-100%) 


PAGE 2 


| On-line Activities** 


How much of your interaction with the instructor occurred online? 


How much of your work involved online group activities (including discussion boards and 
chat)? 


How much of the required work — your assignments — had to be completed online? 


| Interactions with the Instructor 


The instructor in this course really knew me. 
The instructor was active and engaged with the students. 
There was adequate opportunity to interact online with the instructor. ** 


There was adequate opportunity to interact online with professionals in the field. 


| Student Interactions 


I discussed course content with other students. 

Learning activities included meaningful interactions between students in the course.* 
There was adequate opportunity to interact online with other students. ** 

The instructor assigned group projects that required collaborative thinking. 

The instructor connected students with learners from different generations and cultures. 
The instructor inspired students to create virtual learning communities. 

The instructor engaged students in critically analyzing the work of others. 

The instructor provided opportunities for students to create knowledge together. 


Forming an online learning/study group with other students is important to me. *** 


| Student Characteristics 


I believe the online experience was well-suited to the way I like to learn. 
Getting to know other students is important to me. *** 


I believe my course work and grades are secure and private. *** 


I adhere to the university policies and codes of academic honesty as it relates to my assign- 
| ments, discussions, tests, and assessments. *** 


USING ADDITIONAL QUESTIONS FOR ONLINE COURSES 


| Instructor Use of Technology 


The instructor used the technology effectively to communicate the learning objectives. 
The instructor used the technology effectively to engage the students. 


The instructor used the technology effectively to facilitate achievement of the learning 
objectives. 


| Technology and Learning 


I felt I had individualized instruction tailored to my learning needs (able to work at my 
own pace and get help when I needed it). 


I believe the online components for this class were extremely valuable in helping me 


learn. ** 

Asa rule, I work best in self-directed and self-paced course formats. 

The instructional approaches used in this course motivated me to learn. *** 

The use of [insert technology] helped me learn the [insert course material]. 

[Insert teaching method or technology] was a strength of this course. Many of the technology 


Time spent using [insert technology] was productive. items are not under the 
direct control of the 


instructor, but might 


provide useful feedback 


about the learning 
Technology Support experience. 


The [insert resource or technology] was very helpful to me. 

I was able to get technology support when needed. 

Adequate training opportunities were provided to use the technology for this course. 

The technologies [or insert specific technology] used in this course worked the way it was supposed to. 
I was able to understand and follow the course supporting materials (e.g., [insert examples]). 

The communication tools were easy to use (chatroom, message board, e-mail, etc.). 


Thad some problems getting into the course with my assigned password. 
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| Overall Satisfaction 


I would like to take another course that uses [insert technology]. 
I liked the [insert] format of this course. 
I would recommend this kind of class to other students. ** 


All factors considered, the advantages of including online components outweigh the limitations. ** 


Learning Outcomes INSTRUCTIONS 


Using the response options 


Using the Internet for answering questions or solving problems provided, please indicate 
how much progress you 
Summarizing information to guide the learning of others made on each of the 
following: 
Collaboratively creating knowledge with other students 
Learning on my own RESPONSES 
Evaluating the work of other students 1 = No apparent progress 
2= Slight progress 
Writing in a public arena 3 = Moderate progress 
4 = Substantial progress 
Guiding and managing my own learning 5 = Exceptional progress 


| Open-ended/ Free Response 


What aspects of this course contributed most to your learning? 
How could this course be changed to better support your learning? 


What are the advantages and disadvantages of the online environment [or insert specific technologies] 
for your learning in this course? ** 


What suggestions would you offer to the instructor for improvement of this course? 


Tutorial: 


Adding Questions 
in IDEA Online 
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